Smoother C/Rust interop with existing libraries.

[ryankurte]

After attempting to use a collection of c and *-sys libraries in interesting no_std and cross-compiling contexts, it appears to me we have plenty of opportunity to, improve, the state of building and linking existing libraries with rust (and has cost me weeks of my life so far :-/).

While this may not technically be our area of responsibility, I'm not sure where else to put it and I believe it is something that effects us disproportionately (and windows users, trying to compile linked system libraries for windows feels like travelling back in time). If we don't think this should be here (or someone has a better / similar issue) I am happy to close or move the issue elsewhere, but, I think it useful for us to start somewhere ^_^

Also to be clear this is not a critique of existing tooling or the work around this, we have a collection of amazing utilities for use in build.rs and a great ecosystem of crates and bindings. It appears to me that we have a lack of process and abstraction to provide the smooth, packaged, user experience that one hopes for with modern package management, and this is translated to poor publisher experience as it's super difficult to get right, and poor consumer experience as there is no consistency and tooling reliably breaks in all sorts of interesting ways.

Goals

To simplify the construction of *-sys crates that adequately support multiple target architectures, minimising the burden on crate authors to create and manage complex build.rs configurations and improving the crate consumer experience by providing consistent support for a set of required features.

Issues / Requirements / Solutions

I believe most of the issues I have come across could be mitigated through a combination of:

a) the provision of a higher-level c-library integration crate for use in build.rs that abstracts the nuance of building and linking where possible, exposing an opinionated configuration object that manages underlying tools and a set of features to be re-exported by *-sys crates.
b) a spec / guideline for designing *-sys crates with best practices etc.

The following are issues that have arisin during my use of *-sys crates, and some possible mitigations.

*-sys` crates may depend on environmental variables for compilation

This requires a collection of strange environmental variables to be set, on a per-target basis, and is thus difficult to debug or rationalise about for library consumers.

Possible Mitigations

• guide should specify no environmental variables should be required for compilation
• tooling should provide support for per-target-environmental-vars specified in build.rs (and possibly propagated / override from [metadata.PACKAGE][1]

*-sys crates do not build on all (or often, any) no_std or other architectures

This really requires test-builds for all supported architectures

Possible Mitigations

• cross-architecture / target template for building and testing?
• tooling that runs a set of cross-architecture builds?

Pre-built bindgen outputs often do not match target architecture sizes [2] [3]

This results in segfaults and all sorts of other terrible things when running libraries on unexpected architectures

Possible Mitigations

• tooling could build bindgen at compile time, with appropriate target arguments
  • guide should specify that bindgen should be run at compile time
• bindgen could pre-generate a set of variants for different platforms
  • this subset would need to be defined, tested
• packages could default to using pre-packaged bindings, while supporting overrides where required
  • how do we tell whether a binding is valid for a particular architecture in a way that doesn't end in segfaults? do we just need 32-bit and 64-bit versions?

This is likely due to architecture-specific switches in header files, and as such may be too project-dependent for us to provide a good solution :-/

libc does not define types for non tier-1 targets

bindgen defaults to using libc for ffi types, and core::ffi does not contain a number of required types. This means that build scripts often require a modification to use the cty. This often requires patching [4] existing *-sys libraries to detect and alter this behaviour.

Possible Mitigations

• Work is underway to move basic ctypes to the libc repo so the definitions are not empty on unsupported targets [5]
• This may be mitigated by compile-time-bindgen as the correct libc or cty could be selected by target

Package discovery, compilation (and linking) typically doesn't work cross platform

This is a big one, but it's a bit too interrelated to easily split.

*-sys libraries will typically use one mechanism for library discovery, complilation, (and linking), further work is sometimes done to support alternate mechanisms for compilation from source, or static vs. dynamic linking, however, this is often not available and places a huge burden on maintainers to support all the possible variants for a given library.
As an example, using pkg-config which is generally good for unixen tends to not work for windows or static linking (and only sometimes works for cross-compilation under multiarch).

It's also important to be able to link *-sys crates to distro packages for distro packaging, so this probably needs to be specifiable?

Possible Mitigations

• Tooling should be generic over mechanisms for:
  • library / source discovery (pkg-config, vcpkg, local source, fetch from git, curl)
  • compilation (cc, cmake, autoconf etc.)
  • includes (exporting header paths etc.)
  • linking (static, dynamic)

Ideally this would provide a sensible default for your platform, while supporting feature-based overrides as required for different mechanisms for discovery or static compilation etc, as well as configuration overrides from cargo metadata. This way it's reasonably simple to configure per-target build options if required, and to fix build.rs paths etc. from the top level package.

newlib requires function stubs

Many libraries that support embedded compilation via autoconf or other complex mechanism both depend on the standard c libraries and sometimes make a bunch of decisions about linking things like newlib, which then requires function stubs in the rust application.

Possible mitigations

• Support specifying these linker args as part of standard tooling (or, provide improved documentation on the use of .cargo/config for this purpose) (relates to environmental args issue)
• Provide a newlib_stubs and/or libc_stubs that exports a set of c compatible stubs that use the underlying rust allocator etc.

---

If you have come across other issues / solutions, or demonstration of these issues, or have any other thoughts / opinions, please post and I will update the list here ^_^

Other References

• https://users.rust-lang.org/t/improving-the-sys-crate-story/46097
• https://kornel.ski/rust-sys-crate

Existing *-sys library issues

• https://github.com/sodiumoxide/sodiumoxide/issues/377
• Linking on Windows dcuddeback/libusb-rs#20

[genbattle]

My reckons from my (limited) experience with writing sys crates. All of this is fairly generic since I'm not currently working on bare-metal, but I think it applies to no-std and std crates alike.

A lot of the mess around sys crates seems to be a flow-on effect from the poorly defined C and C++ build ecosystem. However, I agree that it should be possible to define a strict interface layer for what these crates expose to the Rust ecosystem, allowing them to abstract over all of the nastyness involved in building C and C++ libraries.

From my perspective there's 3 things that every sys crate should export:

• A minimal rust definition of the functions/types exposed by the library, be it bindgen generated or manually written.
• A search path for the library being wrapped (.so, .lib, .a, etc.).
• A search path for the header files that define the library interface.

Most Rust crates that sit on top of a sys crate will only use the first one, and will just link to the sys crate directly. The other two are useful when using a sys crate to provide a transitive dependency required by another library (e.g. building libjasper requires libjpeg).

The current state of the ecosystem is that external linkage between sys crates is done through environment variables and the cargo:key=value cargo directive. This means that external linkage is done entirely by convention. There's no standard or requirements, and as a result it's inconsistent whether crates provide a root path, a lib directory or an include directory. Some crates provide a root dir, some provide lib and include dirs, others provide some combination of both, or even none.

Bindgen

bindgen is great for getting up and running with an interface, and the bindgen guide is great, I commend the authors of both. If anything the guide needs to be extended further to add a hybrid generate/static configuration where a feature flag is used to trigger generation in the build.rs script only when required and overwrite a static set of bindings in the repository.

This was suggested to me recently on Twitter and I've since decided that it should be the state of the art for my sys crates. It gives the advantage of low build times for systems that follow the general case (e.g. x86_64 systems) while allowing the generation of specific bindings for more exotic systems and use cases.

A separate issue I've seen in sys crates that ties them to a subset of platforms is when libc types (e.g. uint_fast32_t) sneak into the generated bindings. My solution to this so far has been to blacklisting/whitelisting to exclude these types, and then including them from the libc crate so the bindings themselves don't define fixed sizes for these types:

use libc::{size_t, timeval, FILE};

include!("bindings.rs")

However this requires careful grooming by crate maintainers, and is therefore fallible. A better automated solution to prevent C stdlib types from being pulled in (maybe warnings in bindgen?) would be welcome, but I haven't yet come up with anything concrete. Ryan has also mentioned how the libc crate doesn't yet define a complete set of C types, which also hinders this solution in some cases.

Package/library discovery

This is a hard one to solve because ultimately it's platform and library specific. Maybe there's room for a super-crate which abstracts over pkg-config, vcpkg, conan or falls back to a perscribed source build. This would basically be a formalization of the process that I've seen some crates implement in different ways, usually with just one of these package management tools and then a fallback source build (if we're lucky).

[ryankurte]

Thanks for the thoughts!

Ryan has also mentioned how the libc crate doesn't yet define a complete set of C types

A small correction, my experience is that libc does not currently implement any c types on the no_std platforms we're interested in 😅 so allow/blocking types in libc isn't functional anyway. This part of it is however already on the way to resolution so, perhaps one of the smaller concerns.

This is a hard one to solve because ultimately it's platform and library specific. Maybe there's room for a super-crate which abstracts over pkg-config, vcpkg, conan or falls back to a perscribed source build. This would basically be a formalization of the process that I've seen some crates implement in different ways, usually with just one of these package management tools and then a fallback source build (if we're lucky).

This is my conclusion too, though I think we could also integrated the bindgen and search issues we have both referenced into this, and I would hope to both have a reasonable system of fallbacks and to support user overrides for different requirements (like, when we need static libraries because we're on no_std or to compile from source because it's easier than getting the right dlls in the right places on windows).

[kornelski]

I fully agree that configuration of sys crates is a problem. Cargo features are often inappropriate (can't express mutually-exclusive features like static vs dynamic linking), and env vars are non-standard, hard to manage, and undiscoverable.

Sys crates also need a standard way of knowing whether they should export a path to C header files or not. For example, libpng-sys needs zlib-sys to give it .h files.

But this needs to be opt-in, because if crates look for header files by default, they will unnecessarily fail on systems that have only libfoo installed without foo-devel package.

---

Regarding bindgen, I would really advise against running it at compile time. LLVM dependency is a massive pain to install in some environments. It's heavy and adds a lot to compile times.

In my experience bindings generated without platform-specific tests are actually pretty portable. Even in cases when the generated bindings are not that portable, I would still advise against running bindgen at compile time. Crates can pre-generate multiple versions of the bindigs and select right ones at compile time.

---

pkg-config, etc. are pain. But if you're going to replace it with a unified tool, please keep in mind that there may be non-trivial logic involved for selecting which option to use. Such wrapper tool needs to give control to the build script, and not be a black box that pretends it can configure everything using built-in rules.

For example a sys crate may require a specific version of the library, so it may choose to use vendored source code even when pkg-config finds a prebuilt library, but that library turns out to be too old or have a feature disabled. Think how many features and versions ffmpeg's bucket'o'libraries has. Or libjpeg can have 4 different ABIs.

There are snowflake libraries like libpng and llvm that have their own pkg-config derivative. For example, pkg-config definitions for libpng are broken and don't support static linking, only libpng-config does.

[posborne]

Regarding bindgen, I would really advise against running it at compile time. LLVM dependency is a massive pain to install in some environments. It's heavy and adds a lot to compile times.

This has been a point of contention and the way I have been leaning for projects I have interacted with recently. I do think that generating on-the-fly is likely the "right" approach but the requirement on libclang (of a specific version, etc.) seems to push the scales in the direction of pre-generating bindings for different targets.

There has been some discussion about bundling libclang with bindgen which would solve some of these problems, but it of course introduces the problem of building/bundling prebuilt libclang which has its own difficulties, to be sure: rust-lang/rust-bindgen#918. Some combination of these approaches (e.g. prebuilt w/ fallback on dynamic generation using bundled libclang or system libclang) might be the the right balance. This is certainly not an easy thing for all bindgen based *-sys crate maintainers to achieve today but could probably be made easier through some tooling/docs if there is consensus that this is a blessed route.

[fpagliughi]

I've been wrestling with this in the Eclipse Paho MQTT library for a while now. Each release of that crate wraps a very specific version of the Paho C lib, so once you generate the bindings for your target once, you should never need to generate them again.

As @kornelski mentioned for a solution, I tried to pre-generate bindings for as many targets as I could and set build.rs to search for the proper target. That had two problems:

1. Cross-compiling to get the bindings was painful due to the dependencies of the C lib for every target. That left me to natively build bindings on every different target, which takes a lot of time and limits me to the targets I have personally available.

2. When the build can't find the target binding file, it generates an error. The assumption was that this would tell the user to generate their own bindings. The error message, however, is buried so deep in a C build that all it generated was Issue reports on the project GitHub.

[therealprof]

As discussed in todays meeting we don't see anything actionable here yet. Please continue discussion and let us know if you need any assistance.

[therealprof]

Had a quick look in todays meeting, still no news. Will revisit next week.

[ryankurte]

based on my *-sys library experiences i had a bit of a bash at what i imagine something like this might look like, which you can see here, and i've created a bunch of initial issues. i am pretty sure we should be able to abstract a bunch of the complexity from the standard build.rs scripts, but, it's going to be a lot of work to do so...

it seems to me that the next step from here could be to build out the tooling and document best practice, and to test it out against existing *-sys libraries which i imaging will feed back into the tool-building. a bunch of this will involve looking at the build scripts in existing projects and coming up with ways to replicate the required behaviours.

i've tried to summarise how i would expect it to work, though there are probably oversights, and there are a bunch of decisions to be made about defaults and how the whole system works... if you're interested in this approach feel free to hop over there and create issues/prs ^_^

[therealprof]

As discussed in todays meeting this might be worth building a focus group, if you're interest in joining such a group please make yourself known here.

[therealprof]

As discussed in todays meeting this is still a topic which pops up every now and then but it doesn't seem to spike enough interest to form a focus group. As was noted the new package/feature resolver which is supposed to land soon(tm) might actually help a bit with keeping the depedency tree(s) and feature flags straight but it's not quite the requested solution. 😅