Have an option to output warnings from doctests

[Susurrus]

Running cargo test runs doctests by default, and shows warnings for regular code, but not for the doctests themselves. This can be enabled by running cargo test -- --nocapture. I'd like to have warnings also show in our CI tests, even for doctests, but setting -- --nocapture shows all the output which we don't want (and can't control, because they come from the underlying libraries complaining about our tests that are supposed to fail).

I'd think having warnings on for doctests by default would be good, but that wouldn't be backwards compatible. Maybe a --doctest-warnings option could be added in lieu of that?

I believe this was done in 5d145c1, so discussion there might be worth reading. It seems that there are a bunch of warnings that come up with doctests that should be ignored, but there are also ones that can appear that shouldn't be. The question is how do we address both needs?

[GuillaumeGomez]

I can put it back if you want. Just waiting for @rust-lang/docs team confirmation.

[frewsxcv]

This might be a tools team question, less-so a docs team one.

[GuillaumeGomez]

Let's make them join the issue then!

cc @rust-lang/tools

[carols10cents]

I've updated the title to better reflect what I think this request is, please correct it if I've misunderstood.

Many doc tests are small and would have warnings for unused variables, etc. I wouldn't want to see these by default, but I'd be ok with a flag to turn on just output of doc test warnings. I don't think I'd use it, but I wouldn't mind it being there.

Is there any way to get a sense of how many people would use this if it existed and the kinds of problems it would prevent, in order to prioritize this?

[alexcrichton]

The standard library handles this by denying all warnings, but it seems reasonable to have a flag. I don't think we can change the default behavior.

[GuillaumeGomez]

Ok, so I think we can consider this as acceptable. I'll add the flag.

[QuietMisdreavus]

I'm digging into this CLI flag for #49028 and i'm struggling to figure out whether it's useful now. From digging into the implementation that remains, i can find two major features for it:

• Enables --nocapture for doctests, which at the moment just allows it to show compilation warnings
• Turns off the default #![allow(warnings)] that gets applied in regular documentation runs, allowing compilation warnings to be emitted

However, neither one is that useful, because for the first, you can just provide --test-args --nocapture (or -- --nocapture for cargo test) to get functionally the same output, and for the second one, the compilation output seems to still be suppressed anyway. Was there intended to be additional functionality for this?

[QuietMisdreavus]

So here's the results of my testing that led me to this conclusion, and i want to know whether this flag is worth keeping, or whether we need to file additional bugs that can be worked on.

#![crate_type = "lib"]
#![doc(test(attr(warn(warnings))))] // suppress the default allow(unused)

/// (written on a spider's web) Some Struct
///
/// ```
/// let x = 5;
/// println!("sup");
/// ```
pub struct SomeStruct;

struct OtherStruct; // emit a warning when compiling

console output for various interactions with this rust file

$ rustdoc +nightly --version
rustdoc 1.26.0-nightly (392645394 2018-03-15)
$ rustc +nightly g.rs
warning: struct is never used: `OtherStruct`
  --> g.rs:12:1
   |
12 | struct OtherStruct;
   | ^^^^^^^^^^^^^^^^^^^
   |
   = note: #[warn(dead_code)] on by default

$ rustdoc +nightly g.rs
(no output)
$ rustdoc +nightly g.rs -Z unstable-options --display-warnings
(no output)
$ rustdoc +nightly --test g.rs

running 1 test
test g.rs - SomeStruct (line 6) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

$ rustdoc +nightly --test g.rs -Z unstable-options --display-warnings

running 1 test
test g.rs - SomeStruct (line 6) ... ok

successes:

---- g.rs - SomeStruct (line 6) stdout ----
        warning: unused variable: `x`
 --> g.rs:7:5
  |
3 | let x = 5;
  |     ^ help: consider using `_x` instead
  |
  = note: #[warn(unused_variables)] on by default



successes:
    g.rs - SomeStruct (line 6)

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

$ rustdoc +nightly --test g.rs --test-args "--nocapture"

running 1 test
warning: unused variable: `x`
 --> g.rs:7:5
  |
3 | let x = 5;
  |     ^ help: consider using `_x` instead
  |
  = note: #[warn(unused_variables)] on by default

test g.rs - SomeStruct (line 6) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

[GuillaumeGomez]

Should we add an additional option to blockcodes as well? Like:

```rust,show_warnings
let x = 12;
```

[steveklabnik]

Triage: not aware of any movement on this front, though @BurntSushi did come up with a good use-case for this in #55632

[GuillaumeGomez]

Just realized it's been ages ago. We have a --display-warnings option on rustdoc (nightly only for the moment). I'll send a PR for stabilization. Closing this issue then.

EDIT: actually, I think there are issues remaining. Just not sure if they should be considered as issues or not... Re-opening the issue until we have this option stabilized.

[camelid]

Re-opening the issue until we have this option stabilized.

The option is still unstable; the PR that closed this just renamed it.

[jyn514]

Many doc tests are small and would have warnings for unused variables, etc. I wouldn't want to see these by default, but I'd be ok with a flag to turn on just output of doc test warnings. I don't think I'd use it, but I wouldn't mind it being there.

I just opened #91259 which
a) doesn't show any warnings by default,
b) lets you use --test-args --show-output to show all warnings but the unused lints, and
c) lets you add #![warn(unused)] to get all warnings without exceptions.

Does that seem like a reasonable solution? Or do you think rustdoc should show all lints besides unused by default? That's not quite a breaking change, but I'd want to do a crater run to see how noisy it ends up being (especially since I don't think we should add a way to disable the warnings per-invocation if we do that).

[jyn514]

do you think rustdoc should show all lints besides unused by default?

I tried implementing this just now and it was a little tricky, since if rustdoc passes --show-output to libtest, it will show output from all tests, including compile_fail tests. Not sure how this could work.