Add possibility to generate rustdoc JSON output to stdout #128963

GuillaumeGomez · 2024-08-11T09:51:24Z

I think it's likely common to want to get rustdoc json output directly instead of reading it from a file so I added this option to allow it. It's unstable and only works with --output-format=json.

r? @aDotInTheVoid

Urgau · 2024-08-11T10:19:06Z

Why not use the standard -o - combination ?

GuillaumeGomez · 2024-08-11T10:23:55Z

Because I didn't know about it. It's much better!

GuillaumeGomez · 2024-08-11T11:29:43Z

Applied your suggestion @Urgau. Thanks a lot!

aDotInTheVoid

I think this is worth supporting, but if --output - is made to work, then so should --output foo.json

aDotInTheVoid · 2024-08-12T11:10:22Z

src/librustdoc/config.rs

-            (Some(_), Some(_)) => {
-                dcx.fatal("cannot use both 'out-dir' and 'output' at once");
+            (Some(_), Some(output)) => {
+                if output == "-" {


What's the purpose of this logic? rustdoc --out-dir something --output - should be an error, I think.

Because --out-dir is used for all rustdoc-ui tests, preventing this option to be tested. I thought about whether I should add a new flag to tests to allow removing this argument or to allow to have --out-dir and --output - used at the same time and considering it doesn't generate a file, I thought it would be ok. The only thing that I'm not clear about is whether or not it should a warning.

aDotInTheVoid · 2024-08-12T11:11:06Z

src/librustdoc/config.rs

-                dcx.fatal("cannot use both 'out-dir' and 'output' at once");
+            (Some(_), Some(output)) => {
+                if output == "-" {
+                    output_to_stdout = output == "-";


NIT: This is always true by the condition above:

Suggested change

output_to_stdout = output == "-";

output_to_stdout = true;

Oof, silly me. Thanks!

aDotInTheVoid · 2024-08-12T11:16:24Z

tests/rustdoc-ui/json-stdout.stdout

@@ -0,0 +1 @@
+{"root":"0:0:2381","crate_version":null,"includes_private":false,"index":{"0:0:2381":{"id":"0:0:2381","crate_id":0,"name":"json_stdout","span":{"filename":"$DIR/json-stdout.rs","begin":[6,0],"end":[9,15]},"visibility":"public","docs":null,"links":{},"attrs":["#![feature(no_core)]","#![no_core]"],"deprecation":null,"inner":{"module":{"is_crate":true,"items":["0:1:2380"],"is_stripped":false}}},"0:1:2380":{"id":"0:1:2380","crate_id":0,"name":"Foo","span":{"filename":"$DIR/json-stdout.rs","begin":[9,0],"end":[9,15]},"visibility":"public","docs":null,"links":{},"attrs":[],"deprecation":null,"inner":{"struct":{"kind":"unit","generics":{"params":[],"where_predicates":[]},"impls":[]}}}},"paths":{"0:0:2381":{"crate_id":0,"path":["json_stdout"],"kind":"module"},"0:1:2380":{"crate_id":0,"path":["json_stdout","Foo"],"kind":"struct"}},"external_crates":{},"format_version":33}


This test is gonna be super fragile. I think we should use a run-make test instead, that just checks that rustdoc passes, and prints JSON to stdout (maybe with format_version, but no other fields).

That's a great idea and will also solve the --out-dir and --output - options special case.

aDotInTheVoid · 2024-08-12T11:19:19Z

src/librustdoc/json/mod.rs

@@ -37,7 +37,7 @@ pub(crate) struct JsonRenderer<'tcx> {
    /// level field of the JSON blob.
    index: Rc<RefCell<FxHashMap<types::Id, types::Item>>>,
    /// The directory where the blob will be written to.
-    out_path: PathBuf,
+    out_path: Option<PathBuf>,


This should probably be renamed to out_dir, as it's not the path to the JSON file, but the path to the directory that contains the JSON file. (But also, maybe we should do this logic earlier, see my later comments)

Sadly, as we discovered, output and out-dir are actually the same option and output is deprecated. I think it'll require a bigger discussion.

Remark (peanut gallery): that is very surprising, rustc's --output and --out-dir means (I understand this is rustdoc not rustc):

The output filename can be set with the -o flag. A suffix may be added to the filename with the -C extra-filename flag.
Output files are written to the current directory unless the --out-dir flag is used.

cf. https://doc.rust-lang.org/rustc/command-line-arguments.html#--out-dir-directory-to-write-the-output-in

(rustc's cli docs is kinda inaccurate here because it's actually output path for -o not just filename)

See https://github.com/rust-lang/rust/pull/128963/files#r1713682815.

aDotInTheVoid · 2024-08-12T11:20:35Z

src/librustdoc/json/mod.rs

-
-        Ok(())
+            let mut p = out_dir;
+            p.push(output.index.get(&output.root).unwrap().name.clone().unwrap());


We should also respect the --output flag here, even if it's not for a file. Currently --output is ignored for JSON, but if --output - works, then --output some_file.json should also work to rename the JSON (instead of using the crate name).

So, from what I can see, --out-dir and --output are kinda considered the same by rustdoc and --output is deprecated. So currently we don't really have a way to say I want JSON output to be generated into "this file".

Remark (peanut gallery): I would've thought --output=PATH would cause the json output to be dumped to this specific file at PATH, and that if --output is specified, --out-dir would be ignored.

Yes it's the whole problem here. The --output option was created when the HTML format was the only format, making it useless and why we created --out-dir. Currently, both options are actually the same, which is pretty bad.

aDotInTheVoid · 2024-08-12T11:32:30Z

But also, this is complicated by the fact that --output and --out-dir are the same thing for the HTML backend, but this probably shouldn't be the case on the JSON backend.

GuillaumeGomez · 2024-08-12T11:32:35Z

I think this is worth supporting, but if --output - is made to work, then so should --output foo.json

Unless I'm missing something, it's already working.

GuillaumeGomez · 2024-08-12T11:36:17Z

Nevermind, it's not. I'll rewrite the whole PR to make this behaviour better.

rustbot · 2024-08-12T12:22:14Z

The run-make-support library was changed

cc @jieyouxu

This PR modifies tests/run-make/. If this PR is trying to port a Makefile
run-make test to use rmake.rs, please update the
run-make port tracking issue
so we can track our progress. You can either modify the tracking issue
directly, or you can comment on the tracking issue and link this PR.

cc @jieyouxu

GuillaumeGomez · 2024-08-12T12:23:50Z

src/librustdoc/config.rs

            (Some(_), Some(_)) => {
                dcx.fatal("cannot use both 'out-dir' and 'output' at once");
            }
-            (Some(out_dir), None) => out_dir,
-            (None, Some(output)) => output,
+            (Some(out_dir), None) | (None, Some(out_dir)) => {


Since we discovered they're the same option, it simplifies a lot the handling of this new option.

jieyouxu · 2024-08-12T12:31:41Z

src/librustdoc/json/mod.rs

@@ -37,7 +37,7 @@ pub(crate) struct JsonRenderer<'tcx> {
    /// level field of the JSON blob.
    index: Rc<RefCell<FxHashMap<types::Id, types::Item>>>,
    /// The directory where the blob will be written to.
-    out_path: PathBuf,
+    out_path: Option<PathBuf>,


Remark (peanut gallery): that is very surprising, rustc's --output and --out-dir means (I understand this is rustdoc not rustc):

The output filename can be set with the -o flag. A suffix may be added to the filename with the -C extra-filename flag.
Output files are written to the current directory unless the --out-dir flag is used.

cf. https://doc.rust-lang.org/rustc/command-line-arguments.html#--out-dir-directory-to-write-the-output-in

(rustc's cli docs is kinda inaccurate here because it's actually output path for -o not just filename)

jieyouxu · 2024-08-12T12:33:30Z

src/librustdoc/json/mod.rs

-
-        Ok(())
+            let mut p = out_dir;
+            p.push(output.index.get(&output.root).unwrap().name.clone().unwrap());


Remark (peanut gallery): I would've thought --output=PATH would cause the json output to be dumped to this specific file at PATH, and that if --output is specified, --out-dir would be ignored.

tests/run-make/rustdoc-output-stdout/rmake.rs

aDotInTheVoid · 2024-08-14T16:03:25Z

@bors r+

bors · 2024-08-14T16:03:28Z

📌 Commit 63ab7b5 has been approved by aDotInTheVoid

It is now in the queue for this repository.

jieyouxu

The run-make test and library changes LGTM

jieyouxu · 2024-08-14T16:10:38Z

tests/run-make/rustdoc-output-stdout/rmake.rs

+        .arg("-Zunstable-options")
+        .output_format("json")
+        .run()
+        .assert_stdout_contains("{\"");


Remark: I suppose this is a reasonable heuristic.

…iaskrgr Rollup of 8 pull requests Successful merges: - rust-lang#125970 (CommandExt::before_exec: deprecate safety in edition 2024) - rust-lang#127905 (Add powerpc-unknown-linux-muslspe compile target) - rust-lang#128925 (derive(SmartPointer): register helper attributes) - rust-lang#128946 (Hash Ipv*Addr as an integer) - rust-lang#128963 (Add possibility to generate rustdoc JSON output to stdout) - rust-lang#129015 (Update books) - rust-lang#129067 (Use `append` instead of `extend(drain(..))`) - rust-lang#129100 (Fix dependencies cron job) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#128963 - GuillaumeGomez:output-to-stdout, r=aDotInTheVoid Add possibility to generate rustdoc JSON output to stdout Fixes rust-lang#127165. I think it's likely common to want to get rustdoc json output directly instead of reading it from a file so I added this option to allow it. It's unstable and only works with `--output-format=json`. r? `@aDotInTheVoid`

…r=GuillaumeGomez rustdoc-json: Clean up serialization and printing. Somewhat a followup to rust-lang#128963, but makes sense regardless. - Renames `out_path` to `out_dir` because it's not the path to the JSON, but the directory - Also adds a comment explaining `None` - Renames `write` to `serialize_and_write` because it does both. - Also renames the self-profile activity name to be clear this measures both IO cost and serialization CPU cost - Expands the timer to cover flushing - Renames `output` to `output_crate`, to emphasize it's the contents, not the `--output` flag. r? `@GuillaumeGomez`

Rollup merge of rust-lang#129191 - aDotInTheVoid:rdj-serial-cleanup, r=GuillaumeGomez rustdoc-json: Clean up serialization and printing. Somewhat a followup to rust-lang#128963, but makes sense regardless. - Renames `out_path` to `out_dir` because it's not the path to the JSON, but the directory - Also adds a comment explaining `None` - Renames `write` to `serialize_and_write` because it does both. - Also renames the self-profile activity name to be clear this measures both IO cost and serialization CPU cost - Expands the timer to cover flushing - Renames `output` to `output_crate`, to emphasize it's the contents, not the `--output` flag. r? `@GuillaumeGomez`

…jieyouxu Actually parse stdout json, instead of using hacky contains logic. Fixes up the test added in rust-lang#128963, to actually parse the stdout to JSON, instead of just checking that it contains `{"`. CC `@GuillaumeGomez` r? `@jieyouxu`

…jieyouxu Actually parse stdout json, instead of using hacky contains logic. Fixes up the test added in rust-lang#128963, to actually parse the stdout to JSON, instead of just checking that it contains `{"`. CC ``@GuillaumeGomez`` r? ``@jieyouxu``

…jieyouxu Actually parse stdout json, instead of using hacky contains logic. Fixes up the test added in rust-lang#128963, to actually parse the stdout to JSON, instead of just checking that it contains `{"`. CC ```@GuillaumeGomez``` r? ```@jieyouxu```

Rollup merge of rust-lang#129837 - aDotInTheVoid:test-better-json, r=jieyouxu Actually parse stdout json, instead of using hacky contains logic. Fixes up the test added in rust-lang#128963, to actually parse the stdout to JSON, instead of just checking that it contains `{"`. CC ``@GuillaumeGomez`` r? ``@jieyouxu``

Boscop · 2024-10-01T23:19:59Z

@Urgau So how does this work exactly, how can I get the JSON output to stdout? 🙂

When I do cargo +nightly rustdoc -- --output-format json -Z unstable-options -o -, I get:

error: Option 'out-dir' given more than once

Urgau · 2024-10-02T08:29:36Z

cargo already pass --out-dir, and as the error message is saying you can't pass -o/--out-dir multiple times, so you need to bypass Cargo and use rustdoc directly.

Boscop · 2024-10-02T13:10:31Z

@Urgau Thanks for the fast response!

But how can I run rustdoc directly, on the whole workspace, such that it doesn't complain about unresolved imports?
I tried:

> rustdoc -- --output-format json -Z unstable-options -o -
error: too many file operands

For more information about this error, try `rustc --explain E0670`.

Why "error: too many file operands"? 😅

Then I tried:

> rustdoc src/main.rs --edition 2021 --output-format json -Z unstable-options
error[E0432]: unresolved import `google_tasks1`
 --> src/main.rs:1:5
  |
1 | use google_tasks1 as tasks1;
  |     ^^^^^^^^^^^^^^^^^^^^^^^ no external crate `google_tasks1`

error[E0432]: unresolved import `serde`
 --> src/main.rs:2:5
  |
2 | use serde::{Deserialize, Serialize};
  |     ^^^^^ use of undeclared crate or module `serde`

error[E0433]: failed to resolve: use of undeclared crate or module `yup_oauth2`
 --> src/main.rs:9:5
  |
9 | use yup_oauth2::{storage::TokenInfo, InstalledFlowAuthenticator, InstalledFlowReturnMethod};
  |     ^^^^^^^^^^ use of undeclared crate or module `yup_oauth2`

error[E0432]: unresolved import `serde_json`
 --> src/main.rs:3:5
  |
3 | use serde_json::json;
  |     ^^^^^^^^^^ use of undeclared crate or module `serde_json`

error[E0432]: unresolved import `yup_oauth2`
 --> src/main.rs:9:5
  |
9 | use yup_oauth2::{storage::TokenInfo, InstalledFlowAuthenticator, InstalledFlowReturnMethod};
  |     ^^^^^^^^^^ use of undeclared crate or module `yup_oauth2`

error[E0433]: failed to resolve: use of undeclared crate or module `tokio`
  --> src/main.rs:51:3
   |
51 | #[tokio::main]
   |   ^^^^^ use of undeclared crate or module `tokio`

error: aborting due to 6 previous errors

Some errors have detailed explanations: E0432, E0433.
For more information about an error, try `rustc --explain E0432`.

but it complains about unresolved external imports.

Urgau · 2024-10-02T14:33:17Z

I'm sorry but I'm not the author of this change, I don't have time and this discussion is off-topic.
May I suggest you create an issue or use one of our community forum.

rustbot assigned aDotInTheVoid Aug 11, 2024

rustbot added A-rustdoc-json Area: Rustdoc JSON backend S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Aug 11, 2024

GuillaumeGomez force-pushed the output-to-stdout branch from 3868ad5 to caf30e5 Compare August 11, 2024 11:29

GuillaumeGomez changed the title ~~Add --output-to-stdout option to rustdoc~~ Add possibility to generate rustdoc JSON output to stdout Aug 11, 2024

aDotInTheVoid requested changes Aug 12, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 12, 2024

GuillaumeGomez added 2 commits August 12, 2024 13:50

Add possibility to generate rustdoc JSON output to stdout

18d6200

Add read_dir_entries_recursive in run_make_support

e2fd0c0

GuillaumeGomez force-pushed the output-to-stdout branch from caf30e5 to 2b41fbe Compare August 12, 2024 12:22

rustbot added the A-run-make Area: port run-make Makefiles to rmake.rs label Aug 12, 2024

GuillaumeGomez commented Aug 12, 2024

View reviewed changes

GuillaumeGomez requested a review from aDotInTheVoid August 12, 2024 12:24

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 12, 2024

GuillaumeGomez force-pushed the output-to-stdout branch from 2b41fbe to c79c592 Compare August 12, 2024 12:25

jieyouxu reviewed Aug 12, 2024

View reviewed changes

GuillaumeGomez added 2 commits August 12, 2024 14:40

Add run-make test for - for -o option

7882575

Add documentation for -o -

63ab7b5

GuillaumeGomez force-pushed the output-to-stdout branch from c79c592 to 63ab7b5 Compare August 12, 2024 12:40

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 14, 2024

jieyouxu approved these changes Aug 14, 2024

View reviewed changes

matthiaskrgr mentioned this pull request Aug 14, 2024

Rollup of 8 pull requests #129108

Merged

bors merged commit 6c242a0 into rust-lang:master Aug 15, 2024
6 checks passed

rustbot added this to the 1.82.0 milestone Aug 15, 2024

GuillaumeGomez deleted the output-to-stdout branch August 15, 2024 09:21

aDotInTheVoid mentioned this pull request Aug 17, 2024

rustdoc-json: Clean up serialization and printing. #129191

Merged

aDotInTheVoid mentioned this pull request Aug 31, 2024

Actually parse stdout json, instead of using hacky contains logic. #129837

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add possibility to generate rustdoc JSON output to stdout #128963

Add possibility to generate rustdoc JSON output to stdout #128963

GuillaumeGomez commented Aug 11, 2024

Urgau commented Aug 11, 2024

GuillaumeGomez commented Aug 11, 2024

GuillaumeGomez commented Aug 11, 2024

aDotInTheVoid left a comment

aDotInTheVoid Aug 12, 2024

GuillaumeGomez Aug 12, 2024

aDotInTheVoid Aug 12, 2024

GuillaumeGomez Aug 12, 2024

aDotInTheVoid Aug 12, 2024

GuillaumeGomez Aug 12, 2024

aDotInTheVoid Aug 12, 2024

GuillaumeGomez Aug 12, 2024

jieyouxu Aug 12, 2024 •

edited

Loading

GuillaumeGomez Aug 12, 2024

aDotInTheVoid Aug 12, 2024

GuillaumeGomez Aug 12, 2024

GuillaumeGomez Aug 12, 2024

jieyouxu Aug 12, 2024

GuillaumeGomez Aug 12, 2024

aDotInTheVoid commented Aug 12, 2024

GuillaumeGomez commented Aug 12, 2024

GuillaumeGomez commented Aug 12, 2024

rustbot commented Aug 12, 2024

GuillaumeGomez Aug 12, 2024

jieyouxu Aug 12, 2024 •

edited

Loading

jieyouxu Aug 12, 2024

aDotInTheVoid commented Aug 14, 2024

bors commented Aug 14, 2024

jieyouxu left a comment

jieyouxu Aug 14, 2024

Boscop commented Oct 1, 2024

Urgau commented Oct 2, 2024

Boscop commented Oct 2, 2024

Urgau commented Oct 2, 2024

		@@ -0,0 +1 @@
		{"root":"0:0:2381","crate_version":null,"includes_private":false,"index":{"0:0:2381":{"id":"0:0:2381","crate_id":0,"name":"json_stdout","span":{"filename":"$DIR/json-stdout.rs","begin":[6,0],"end":[9,15]},"visibility":"public","docs":null,"links":{},"attrs":["#![feature(no_core)]","#![no_core]"],"deprecation":null,"inner":{"module":{"is_crate":true,"items":["0:1:2380"],"is_stripped":false}}},"0:1:2380":{"id":"0:1:2380","crate_id":0,"name":"Foo","span":{"filename":"$DIR/json-stdout.rs","begin":[9,0],"end":[9,15]},"visibility":"public","docs":null,"links":{},"attrs":[],"deprecation":null,"inner":{"struct":{"kind":"unit","generics":{"params":[],"where_predicates":[]},"impls":[]}}}},"paths":{"0:0:2381":{"crate_id":0,"path":["json_stdout"],"kind":"module"},"0:1:2380":{"crate_id":0,"path":["json_stdout","Foo"],"kind":"struct"}},"external_crates":{},"format_version":33}

Add possibility to generate rustdoc JSON output to stdout #128963

Add possibility to generate rustdoc JSON output to stdout #128963

Conversation

GuillaumeGomez commented Aug 11, 2024

Urgau commented Aug 11, 2024

GuillaumeGomez commented Aug 11, 2024

GuillaumeGomez commented Aug 11, 2024

aDotInTheVoid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jieyouxu Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aDotInTheVoid commented Aug 12, 2024

GuillaumeGomez commented Aug 12, 2024

GuillaumeGomez commented Aug 12, 2024

rustbot commented Aug 12, 2024

Choose a reason for hiding this comment

jieyouxu Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aDotInTheVoid commented Aug 14, 2024

bors commented Aug 14, 2024

jieyouxu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Boscop commented Oct 1, 2024

Urgau commented Oct 2, 2024

Boscop commented Oct 2, 2024

Urgau commented Oct 2, 2024

jieyouxu Aug 12, 2024 •

edited

Loading

jieyouxu Aug 12, 2024 •

edited

Loading