Tracking Issue for Cross Compiling Doctests

[Goirad]

This is an issue to track the functionality added by PR#60387, and also by extension this PR in cargo.

PR#60387 adds three options to rustdoc:

• --enable-per-target-ignores which adds a compiletest-like mechanism for ignoring platform on a substring basis (eg ignore-foo)
• --runtool and --runtool-arg for specifying an external program and its arguments, to which the compiled doctest binary will be passed as the final argument

The companion PR adds a flag to cargo for enabling this feature as well as cross-compiling doctests and parsing runtool tokens from a .cargo/config.

Eventually another PR for x.py can either enable this unconditionally or add a flag for testing the extensive documentation in the standard library on other platforms.

This solves cargo issue #6460

[jyn514]

@Goirad I don't quite understand - what is this tracking? It sounds like all the functionality is already implemented in rustdoc, what's left to do?

[ehuss]

@jyn514 The rustdoc flags are still unstable, so this is tracking the stabilization of those flags.

[safinaskar]

My host is x86_64-unknown-linux-gnu. I'm building for x86_64-unknown-linux-musl. Resulting binaries can be run on this host without any problems. But cargo test --target=x86_64-unknown-linux-musl -Zdoctest-xcompile simply compiles test instead of running them. Moreover, this command doesn't even warn that doctests was merely compiled instead of run! I want tests to run. And I don't want some kind of Cargo.toml configuring for this.

Moreover, if I don't add -Zdoctest-xcompile, doctests are not compiled at all, and there is no any warning about this

[ehuss]

@safinaskar Can you provide some more information? The tests appear to be run for me with cross compiling to x86_64-unknown-linux-musl:

~/Temp/foo> cargo test --target x86_64-unknown-linux-musl -Zdoctest-xcompile
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests (target/x86_64-unknown-linux-musl/debug/deps/foo-65396ef18f7af6ca)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

   Doc-tests foo

running 1 test
test src/lib.rs - abc (line 1) ... FAILED

failures:

---- src/lib.rs - abc (line 1) stdout ----
Test executable failed (exit code 101).

stderr:
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `2`', src/lib.rs:3:1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

As for warnings, there is a warning displayed when the verbose flag:

~/Temp/foo> cargo test --target x86_64-unknown-linux-musl --doc -v
       Fresh foo v0.1.0 (/home/eric/Temp/foo)
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
note: skipping doctests for foo v0.1.0 (/home/eric/Temp/foo) (lib), cross-compilation doctests are not yet supported
See https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#doctest-xcompile for more information.

The reason it is only displayed with -v is that otherwise it would be displayed 100% of the time when cross-compiling, and there is no workaround other than using nightly. We felt that would be an over-agressive warning for now.

We definitely recognize that can cause people to not know that tests are not being run. Hopefully most of the time that is evident by the absence of any doc-test display.

I would like to see this feature move towards stabilization to help prevent that oversight, so any experience reports would be helpful if you get it to work.

[safinaskar]

@ehuss

Can you provide some more information?

I am unable to reproduce. I. e. now everything works. It seems there was no problem in the first place, sorry

[kpfleming]

For what it's worth, we've been using this unstable feature at Fastly for most of this year (2024), compiling a moderate number of code examples during every CI run for one of our documentation repositories (the target is wasm32-wasi). We haven't had any issues with it, it works well for us.

[ehuss]

Just an update, I had a small discussion about this at https://rust-lang.zulipchat.com/#narrow/stream/266220-t-rustdoc/topic/stabilizing.20doctest.20xcompile. It sounds like there are a few minor things to decide:

• Bikeshed the name of --runtool and --runtool-arg. @Nemo157 suggested --test-runtool to match the style of --test-builder.
• When stabilizing, should we just drop --enable-per-target-ignores and make it always enabled? That sounds good to me.

The big question is if we are OK with potentially breaking projects that are doing cargo test --target …, where their doctests don't work on that target. I tried this on the standard library, and identified that arm-android failed on some tests. They look easy to fix, but are we OK with that in general? Personally I'm OK with it as long as we have some very clear notices in the release notes and blog post.

[weihanglo]

Instead of stabilizing these rustdoc flags, if Cargo could take over the test execution (which is prposed in rust-lang/testing-devex-team#5), it is more nature to use. And rustdoc may not need to maintain those flags?
(If rustdoc team is happy to stabilize these flags, I am okay as well)

That said, it doesn't resolve doctest failures ehuss mentioned :(