Compilation fails on build dependency, Cannot bind to C library.

[Tooney712]

I'm not sure if this should be here, in Regressions, or Library Tracking Issue, but regardless... I can't get a package dependency (dav1d-rs) to compile when running cargo check in Rust nightly. If I try to build it in Rust stable, it works perfectly fine and builds with no errors. As far as I can tell, there's no issue with the package's build script, it seems to be Bindgen that is throwing a fit(?).

I'm including the relevant code from the package in question, but there doesn't seem to be any problems with it.

Code

Link to dav1d-sys build.rs

build.rs

use std::env;
use std::path::PathBuf;

mod build {
    use super::*;
    use std::path::Path;
    use std::process::{Command, Stdio};

    const REPO: &str = "https://code.videolan.org/videolan/dav1d.git";
    const TAG: &str = "1.3.0";

    macro_rules! runner {
        ($cmd:expr, $($arg:expr),*) => {
            Command::new($cmd)
                $(.arg($arg))*
                .stderr(Stdio::inherit())
                .stdout(Stdio::inherit())
                .output()
                .expect(concat!($cmd, " failed"));

        };
    }

    pub fn build_from_src(
        lib: &str,
        _version: &str,
    ) -> Result<system_deps::Library, system_deps::BuildInternalClosureError> {
        let build_dir = "build";
        let release_dir = "release";

        let source = PathBuf::from(env::var("OUT_DIR").unwrap()).join("dav1d");
        let build_path = source.join(build_dir);
        let release_path = source.join(release_dir);

        if !Path::new(&source.join(".git")).exists() {
            runner!("git", "clone", "--depth", "1", "-b", TAG, REPO, &source);
        } else {
            runner!(
                "git",
                "-C",
                source.to_str().unwrap(),
                "fetch",
                "--depth",
                "1",
                "origin",
                TAG
            );
            runner!(
                "git",
                "-C",
                source.to_str().unwrap(),
                "checkout",
                "FETCH_HEAD"
            );
        }

        runner!(
            "meson",
            "setup",
            "-Ddefault_library=static",
            "--prefix",
            release_path.to_str().unwrap(),
            build_path.to_str().unwrap(),
            source.to_str().unwrap()
        );
        runner!("ninja", "-C", build_path.to_str().unwrap());
        runner!("meson", "install", "-C", build_path.to_str().unwrap());

        let pkg_dir = build_path.join("meson-private");
        system_deps::Library::from_internal_pkg_config(&pkg_dir, lib, TAG)
    }
}

fn main() {
    if std::env::var("DOCS_RS").is_ok() {
        return;
    }

    system_deps::Config::new()
        .add_build_internal("dav1d", build::build_from_src)
        .probe()
        .unwrap();
}

From lib.rs

#[repr(C)]
#[derive(Copy, Clone)]
pub struct Dav1dWarpedMotionParams {
    pub type_: Dav1dWarpedMotionType,
    pub matrix: [i32; 6usize],
    pub u: Dav1dWarpedMotionParamsU,
}

#[repr(C)]
#[derive(Copy, Clone)]
pub union Dav1dWarpedMotionParamsU {
    pub p: Dav1dWarpedMotionParamsUP,
    pub abcd: [i16; 4usize],
}

#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct Dav1dWarpedMotionParamsUP {
    pub alpha: i16,
    pub beta: i16,
    pub gamma: i16,
    pub delta: i16,
}

From dav1d/include/dav1d/headers.h

enum Dav1dWarpedMotionType {
    DAV1D_WM_TYPE_IDENTITY,
    DAV1D_WM_TYPE_TRANSLATION,
    DAV1D_WM_TYPE_ROT_ZOOM,
    DAV1D_WM_TYPE_AFFINE,
};

typedef struct Dav1dWarpedMotionParams {
    enum Dav1dWarpedMotionType type;
    int32_t matrix[6];
    union {
        struct {
            int16_t alpha, beta, gamma, delta;
        } p;
        int16_t abcd[4];
    } u;
} Dav1dWarpedMotionParams;

Meta

rustc --version --verbose:

rustc 1.77.0-nightly (d78329b92 2024-01-13)
binary: rustc
commit-hash: d78329b92e8d141d19505e7c1527181c4ab87ed4
commit-date: 2024-01-13
host: x86-64-unknown-linux-gnu
release: 1.77.0-nightly
LLVM version: 17.0.6

I checked this in both nightly 2024-01-11 and 2024-01-13, the result was the same for both.

Error output

  Compiling dav1d-sys v0.3.5
error: failed to run custom build command for `dav1d-sys v0.3.5`

Caused by:
  process didn't exit successfully: `$(PROJECTDIR)/target/debug/build/dav1d-sys-2befe1be2c51b161/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=DAV1D_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=SYSROOT
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rustc-link-lib=dav1d
  cargo:include=/usr/include
  cargo:rerun-if-env-changed=SYSTEM_DEPS_BUILD_INTERNAL
  cargo:rerun-if-env-changed=SYSTEM_DEPS_LINK
  cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_LIB
  cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_LIB_FRAMEWORK
  cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_SEARCH_NATIVE
  cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_SEARCH_FRAMEWORK
  cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_INCLUDE
  cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_BUILD_INTERNAL
  cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_LINK

  cargo:rustc-cfg=system_deps_have_dav1d

  --- stderr
  thread 'main' panicked at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/ir/context.rs:878:9:
  "Dav1dWarpedMotionParams_union_(unnamed_at_/usr/include/dav1d/headers_h_100_5)" is not a valid Ident
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Included CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true as dav1d-sys is a build dependency for dav1d-rs.

Backtrace

  stack backtrace:
     0: rust_begin_unwind
               at /rustc/d78329b92e8d141d19505e7c1527181c4ab87ed4/library/std/src/panicking.rs:647:5
     1: core::panicking::panic_fmt
               at /rustc/d78329b92e8d141d19505e7c1527181c4ab87ed4/library/core/src/panicking.rs:72:14
     2: proc_macro2::fallback::validate_ident
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.71/src/fallback.rs:828:9
     3: proc_macro2::fallback::Ident::new_checked
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.71/src/fallback.rs:760:9
     4: proc_macro2::imp::Ident::new_checked
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.71/src/wrapper.rs:646:50
     5: proc_macro2::Ident::new
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/proc-macro2-1.0.71/src/lib.rs:956:21
     6: bindgen::ir::context::BindgenContext::rust_ident_raw
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/ir/context.rs:878:9
     7: bindgen::ir::context::BindgenContext::rust_ident
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/ir/context.rs:870:9
     8: <bindgen::ir::comp::CompInfo as bindgen::codegen::CodeGenerator>::codegen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:1731:31
     9: <bindgen::ir::ty::Type as bindgen::codegen::CodeGenerator>::codegen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:806:39
    10: <bindgen::ir::item::Item as bindgen::codegen::CodeGenerator>::codegen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:492:17
    11: <bindgen::ir::comp::CompInfo as bindgen::codegen::CodeGenerator>::codegen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:2062:13
    12: <bindgen::ir::ty::Type as bindgen::codegen::CodeGenerator>::codegen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:806:39
    13: <bindgen::ir::item::Item as bindgen::codegen::CodeGenerator>::codegen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:492:17
    14: <bindgen::ir::module::Module as bindgen::codegen::CodeGenerator>::codegen::{{closure}}
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:515:21
    15: <bindgen::ir::module::Module as bindgen::codegen::CodeGenerator>::codegen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:545:13
    16: <bindgen::ir::item::Item as bindgen::codegen::CodeGenerator>::codegen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:483:17
    17: bindgen::codegen::codegen::{{closure}}
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:4287:9
    18: bindgen::ir::context::BindgenContext::gen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/ir/context.rs:1190:19
    19: bindgen::codegen::codegen
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/codegen/mod.rs:4251:5
    20: bindgen::Bindings::generate
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/lib.rs:2374:32
    21: bindgen::Builder::generate
               at $(HOME)/.cargo/registry/src/index.crates.io-6f17d22bba15001f/bindgen-0.59.2/src/lib.rs:1478:9
    22: build_script_build::main
               at ./build.rs:77:13
    23: core::ops::function::FnOnce::call_once
               at /rustc/d78329b92e8d141d19505e7c1527181c4ab87ed4/library/core/src/ops/function.rs:250:5
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[Noratrieb]

Thank you for the report! It would be useful to bisect the regression using cargo-bisect-rustc to make it easier to figure out what the problem could be.

[Noratrieb]

I tried to reproduce this with the following script in a fresh podman run -it docker.io/rust /bin/bash but was not able to reproduce any issues, it all worked and checked successfully. (copied the setup code from dav1d-rs's CI)

rustup toolchain install nightly-2024-01-13
rustup default nightly-2024-01-13

apt update
apt install -y meson ninja-build nasm
git clone https://github.com/rust-av/dav1d-rs.git
cd dav1d-rs
git checkout a73672d040b6a8702115fcb00c697104576ee8f5

export DAV1D_DIR=dav1d_dir
export LIB_PATH=lib/x86_64-linux-gnu

git clone --branch 1.3.0 --depth 1 https://code.videolan.org/videolan/dav1d.git
cd dav1d
git checkout 48035599cdd4e4415732e408c407e0c1cd1c7444 # If this fails, do more depth
meson build -Dprefix=$HOME/$DAV1D_DIR -Denable_tools=false -Denable_examples=false --buildtype release

ninja -C build
ninja -C build install

export PKG_CONFIG_PATH="$HOME/$DAV1D_DIR/$LIB_PATH/pkgconfig"
export LD_LIBRARY_PATH="$HOME/$DAV1D_DIR/$LIB_PATH"

cd ..
cargo check

[Noratrieb]

What's your bindgen version in your lockfile?
While I cannot reproduce this, the error message looks like it might be a result of someone relying on token pretty printing which was changed in #114571. cc @nnethercote are you aware of any known issues here with bindgen?

[Tooney712]

The bindgen version mentioned in the lockfile is 0.59.2. I also have a separate bindgen installed in .cargo/bin/ with version 0.66.1. I ran cargo-bisect-rustc up to nightly-2023-03-04 before stopping it, it failed every time.

That having been said, I found that the dav1d-rs crate is being called as a dependency of the image crate, which I didn't realize when writing up the issue. Looking into image's features shows that the dav1d dependency only works on stable rust. I don't particularly know why it only works on stable rust, but regardless this seems to be more of an issue of me being stupid and not fully investigating the error. I apologize for the disruption, hope you have a good day.

[Noratrieb]

Looking into image's features shows that the dav1d dependency only works on stable rust.

That does not sound good and is definitely a bug somewhere..., I'll reopen the issue until we know that it's a bug in a crate and not in rustc.

[Noratrieb]

It you could figure out why nightly breaks, that would be very useful :)
(Also, no worries about opening this issue, it is definitely extremely reasonable to expect things to work on nightly just fine without reading any documentation)

[bjorn3]

Which LLVM version do you have installed on your system? rust-bindgen uses it's libclang library for parsing C header files.

[Tooney712]

Apologies for responding to this so late, I didn't check GitHub for a couple of days. The LLVM version is 16.0.6 from the Arch Linux repo (which I'm guessing is based on 16.0.4 since that's the highest version in the 16 series on llvm.org).

In other unfortunate news, I can't get the bug to happen anymore. I did do a package upgrade for dependencies, but looking at the changelogs that shouldn't have changed anything code-wise, image updated their docs but didn't touch any other part of their code. I tried recreating the problem with a fresh project, making sure to keep the same version numbers on the packages from before I upgraded them, but that built just fine. I have zero explanation, and no idea as to how I could recreate the problem now.

[nnethercote]

cc @nnethercote are you aware of any known issues here with bindgen?

Not with bindgen. I'm only aware of some rare breakage in proc macros that use .to_string() on the token stream and then do regex/substring matching on the output (which is totally not the right way to write a proc macro).

[Freddalf]

I'm having the same issue while trying to build gst-plugin-rs https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs as a module of proton.
I use the stable version of rust provided from Archlinux repos (1.75.0) and LLVM (16.0.6)
The generated lockfile that I believe is created (but being built as part of a larger package and it is hard for me to verify what is being pulled and moved) uses the old and similarly dated version of dav1d-sys as Tooney (0.3.4 in my case).
The tagged git-version I'm cloning has version 0.7.3 of dav1d-sys in its lockfile, but the lockfile is not cloned as it is specified to be ignored by git.
The key differences looking at the lockfiles seems to indicate that, what I will call the 'outdated' crates, are all from roughly the same timeframe (spring 2021).
What can cause this?
Tooney, do you still have a lockfile from when it did compile so that you can verify that it compiled using a newer version of the crate?
Also, the bindgen package from my generated lockfile is 0.58.1.
Thanks!