Implementing "<test_binary> --list --format json" for use by IDE test explorers / runners #108148

parthopdas · 2023-02-17T00:56:44Z

Fixes #107307

PR 1 of 2 - wiring up just the new information + implement the command line changes i.e. --format json + tests

upcoming:
PR 2 of 2 - clean up "#[cfg(not(bootstrap))]" from PR 1

As per the discussions on

rustbot · 2023-02-17T00:56:51Z

r? @cuviper

(rustbot has picked a reviewer for you, use r? to override)

rustbot · 2023-02-17T00:56:55Z

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

Stabilizing library features
Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
Changing public documentation in ways that create new stability guarantees
Changing observable runtime behavior of library APIs

parthopdas · 2023-02-17T00:57:28Z

@Manishearth fyi

Manishearth · 2023-02-17T02:01:52Z

@cuviper For background: The plan here only affects a preexisting unstable CLI option (--format), so it should be possible to make these changes without worrying about backcompat. The author is planning on also making a proper RFC for the feature after this lands based on the existing implementation.

(though the TestDescAndFn changes are going to be a pain for nightly compiletest, oh well)

Manishearth · 2023-02-17T02:02:43Z

Also, I do not know enough about TestDescAndFn to have a good opinion on whether this is the right way to make this change, @oli-obk or @cuviper might.

parthopdas · 2023-02-17T02:33:46Z

(though the TestDescAndFn changes are going to be a pain for nightly compiletest, oh well)

@Manishearth this PR already contains changes to a bunch of tests. but if your comment refers to something else i am happy to fix them if you let me know how to find them.

compiler/rustc_span/src/source_map.rs

bjorn3 · 2023-02-17T11:01:32Z

compiler/rustc_builtin_macros/src/test.rs

+                                    // end_line: end line of the test fn identifier.
+                                    field("end_line", cx.expr_usize(sp, location_info.3)),
+                                    // end_col: end column of the test fn identifier.
+                                    field("end_col", cx.expr_usize(sp, location_info.4)),


Is the end location really useful?

it is my guess right now, i think i will be needing this for the UI adornments e.g. highlighting the symbol when clicked on the test explorer and adding code lens related glyphs.

ps: pretty sure responded to this. not sure why my comments got reset.

compiler/rustc_span/src/source_map.rs

bjorn3 · 2023-02-17T11:07:17Z

compiler/rustc_builtin_macros/src/test.rs

@@ -231,6 +231,8 @@ pub fn expand_test_or_bench(
        &item.ident,
    ));

+    let location_info = cx.sess.source_map().span_to_locatio_info(item.ident.span);


Maybe do

if sess.should_collapse_debuginfo(span) { // Walk up the macro expansion chain until we reach a non-expanded span. // We also stop at the function body level because no line stepping can occur // at the level above that. // Use span of the outermost expansion site, while keeping the original lexical scope. span = rustc_span::hygiene::walk_chain(span, self.mir.span.ctxt()); }

to get to the macro invocation site rather than macro definition site? The same is done for debuginfo:

rust/compiler/rustc_codegen_ssa/src/mir/debuginfo.rs

Lines 210 to 216 in 9556b56

if self.cx.tcx().should_collapse_debuginfo(span) {

// Walk up the macro expansion chain until we reach a non-expanded span.

// We also stop at the function body level because no line stepping can occur

// at the level above that.

// Use span of the outermost expansion site, while keeping the original lexical scope.

span = rustc_span::hygiene::walk_chain(span, self.mir.span.ctxt());

}

this is a good suggestion, i think.

sess is not available in source_span, so i added it on test.rs line 234 (the one you highlighted above). however there "self.mir" is not available. any ideas? sess here is ExtCtxt while in debuginfo.rs self is FunctionCtx.

Good question. Didn't notice the self.mir. I'm not sure how to replace it.

i based my changes on span_to_string in source_map.rs. also searched unsuccessfully for a way to do what you recommended in there and in tests.rs.

for now i'll close it unless someone can help out.

Manishearth · 2023-02-17T15:01:01Z

note on commit messages: there's no need to put the issue number at the beginning of each message, and you probably don't want commits to have the same message: have each one describe what it does

(right now when looking at a list of commits you don't really see anything useful because they're too long and overlapping)

oli-obk · 2023-02-17T15:43:47Z

Just for visibility, this should get an MCP, I don't see it being controversial, but compiler folks should know about it via that mechanism.

parthopdas · 2023-02-17T17:23:44Z

(right now when looking at a list of commits you don't really see anything useful because they're too long and overlapping)

oh but they do - just need to expand the description

parthopdas · 2023-02-17T17:24:40Z

drats! closed the PR by mistake. @#!$@$!!!

Manishearth · 2023-02-17T17:25:10Z

oh but they do - just need to expand the description

Yeah, I mean there's no need for the commit header to be the PR title for each one; just have your short message be the main commit message

parthopdas · 2023-02-17T17:26:01Z

Just for visibility, this should get an MCP, I don't see it being controversial, but compiler folks should know about it via that mechanism.

@oli-obk please point me to the process - happy to do that.

parthopdas · 2023-02-17T23:54:58Z

@oli-obk @bjorn3 i am pretty sure i responded to all the reviews in the morning but for some reason they are missing. responded to them again just now. let me know what you think.

parthopdas · 2023-02-19T16:21:17Z

@oli-obk raised the MCP https://rust-lang.zulipchat.com/#narrow/stream/233931-t-compiler.2Fmajor-changes/topic/Implementing.20.22.3Ctest_binary.3E.20--list.20--form.E2.80.A6.20compiler-team.23592/near/328747548

parthopdas · 2023-03-03T02:49:58Z

@oli-obk please can you approve this PR?

compiler/rustc_span/src/source_map.rs

oli-obk · 2023-03-03T16:41:00Z

Please rebase the merge commits away and squash your fixup commits (or just make one big commit, that's also ok).

Do we have documentation somewhere for the existing json output of the test suite?

oli-obk · 2023-03-16T05:57:47Z

@bors r+

bors · 2023-03-16T05:57:49Z

📌 Commit 3720753 has been approved by oli-obk

It is now in the queue for this repository.

Implementing "<test_binary> --list --format json" for use by IDE test explorers / runners Fixes rust-lang#107307 PR 1 of 2 - wiring up just the new information + implement the command line changes i.e. --format json + tests upcoming: PR 2 of 2 - clean up "#[cfg(not(bootstrap))]" from PR 1 As per the discussions on - MCP: https://rust-lang.zulipchat.com/#narrow/stream/233931-t-compiler.2Fmajor-changes/topic/Implementing.20.22.3Ctest_binary.3E.20--list.20--form.E2.80.A6.20compiler-team.23592/near/328747548 - preRFC: https://internals.rust-lang.org/t/pre-rfc-implementing-test-binary-list-format-json-for-use-by-ide-test-explorers-runners/18308 - FYI on Discord: https://discord.com/channels/442252698964721669/459149169546887178/1075581549409484820

Manishearth · 2023-03-18T18:54:07Z

@bors rollup=iffy

bors · 2023-03-18T21:33:15Z

⌛ Testing commit 3720753 with merge 0d88c5e8fdfd901740e353a249f17c1f8cf9d9c2...

bors · 2023-03-18T21:40:14Z

💔 Test failed - checks-actions

rust-log-analyzer · 2023-03-18T21:42:59Z

The job wasm32 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

Building stage1 library artifacts (x86_64-unknown-linux-gnu) 
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc - --crate-name ___ --print=file-names -Csymbol-mangling-version=legacy -Zunstable-options -Zunstable-options '--check-cfg=values(bootstrap)' '--check-cfg=values(stdarch_intel_sde)' '--check-cfg=values(no_fp_fmt_parse)' '--check-cfg=values(no_global_oom_handling)' '--check-cfg=values(no_rc)' '--check-cfg=values(no_sync)' '--check-cfg=values(freebsd12)' '--check-cfg=values(backtrace_in_libstd)' '--check-cfg=values(target_env,"libnx")' '--check-cfg=values(target_arch,"asmjs","spirv","nvptx","xtensa")' -Zmacro-backtrace -Clink-args=-Wl,-z,origin '-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Csplit-debuginfo=off -Cprefer-dynamic -Cembed-bitcode=yes '-Zcrate-attr=doc(html_root_url="https://doc.rust-lang.org/nightly/")' --target x86_64-unknown-linux-gnu --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 254)
  --- stderr
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(+0x97e043)[0x7fdea6a8e043]
  /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7fdea5d9b090]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-16-rust-1.70.0-nightly.so(+0x7262e06)[0x7fdea4638e06]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(+0xd0fa64)[0x7fdea6e1fa64]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(+0xbca65c)[0x7fdea6cda65c]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(+0xbc9794)[0x7fdea6cd9794]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(_RNvXs5_Cs7DTEPoQIU0M_18rustc_codegen_llvmNtB5_18LlvmCodegenBackendNtNtNtCsiydnvZ3aRh1_17rustc_codegen_ssa6traits7backend14CodegenBackend15target_features+0x1b)[0x7fdea6d397cb]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(_RNvNtCsbMR7lNZiyEF_15rustc_interface4util17add_configuration+0x37)[0x7fdea6b7b887]
Build completed unsuccessfully in 0:03:20
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(_RNvNtCsbMR7lNZiyEF_15rustc_interface4util14create_session+0x89d)[0x7fdea6b7c32d]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(+0x988590)[0x7fdea6a98590]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(+0x983310)[0x7fdea6a93310]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-fdca23ff32bf7124.so(+0x984e53)[0x7fdea6a94e53]
  /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/libstd-2268eb07c4da6a07.so(rust_metadata_std_e292dd3bffb96032+0xca4ae)[0x7fdea60184ae]
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7fde9d1b6609]
  /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fdea5e77133]
  [RUSTC-TIMING] ___ test:false 0.203
  rustc exited with signal: 11 (SIGSEGV) (core dumped)

ehuss · 2023-03-18T22:02:31Z

@bors retry
wasm SIGSEGV

parthopdas · 2023-03-18T22:06:17Z

bors CI failure, tree closed now. updates here https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/segfaults/near/342843524

bors · 2023-03-20T03:24:30Z

⌛ Testing commit 3720753 with merge 9d0eac4...

bors · 2023-03-20T05:56:23Z

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing 9d0eac4 to master...

rust-timer · 2023-03-20T08:53:48Z

Finished benchmarking commit (9d0eac4): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.4%	[0.4%, 0.4%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.4%	[0.4%, 0.4%]	1

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.3%	[0.5%, 4.0%]	6
Regressions ❌ (secondary)	2.7%	[1.2%, 5.0%]	74
Improvements ✅ (primary)	-1.6%	[-1.6%, -1.6%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.8%	[-1.6%, 4.0%]	7

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.5%	[0.4%, 0.6%]	4
Regressions ❌ (secondary)	2.1%	[2.1%, 2.1%]	1
Improvements ✅ (primary)	-1.2%	[-1.2%, -1.2%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[-1.2%, 0.6%]	5

Update the toolchain to use nightly-2023-04-16. Changes were related to the following changes to the toolchain: - rust-lang/rust#108944 - rust-lang/rust#108148 - rust-lang/rust#108471 - rust-lang/rust#109358 - rust-lang/rust#109849 - rust-lang/rust#109819 - rust-lang/rust#109718 - rust-lang/rust#109092 - rust-lang/rust#108856 - rust-lang/rust#105613 - rust-lang/rust#103042 - rust-lang/rust#109716 - rust-lang/rust#108340 - rust-lang/rust#102906 - rust-lang/rust#98112 - rust-lang/rust#108080

rustbot assigned cuviper Feb 17, 2023

bjorn3 reviewed Feb 17, 2023

View reviewed changes

compiler/rustc_span/src/source_map.rs Outdated Show resolved Hide resolved

bjorn3 reviewed Feb 17, 2023

View reviewed changes

compiler/rustc_span/src/source_map.rs Show resolved Hide resolved

bjorn3 reviewed Feb 17, 2023

View reviewed changes

parthopdas closed this Feb 17, 2023

parthopdas reopened this Feb 17, 2023

parthopdas mentioned this pull request Mar 2, 2023

Implementing "<test_binary> --list --format json" for use by IDE test explorers / runners rust-lang/compiler-team#592

Closed

oli-obk requested changes Mar 3, 2023

View reviewed changes

compiler/rustc_span/src/source_map.rs Show resolved Hide resolved

compiler/rustc_span/src/source_map.rs Outdated Show resolved Hide resolved

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 16, 2023

matthiaskrgr mentioned this pull request Mar 17, 2023

Rollup of 7 pull requests #109285

Closed

matthiaskrgr mentioned this pull request Mar 18, 2023

Rollup of 8 pull requests #109303

Merged

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 18, 2023

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 18, 2023

bors added the merged-by-bors This PR was explicitly merged by bors. label Mar 20, 2023

bors merged commit 9d0eac4 into rust-lang:master Mar 20, 2023

rustbot added this to the 1.70.0 milestone Mar 20, 2023

rpwachowski mentioned this pull request Mar 26, 2023

Set test case function Ident spans to corresponding test_case spans frondeus/test-case#118

Closed

celinval mentioned this pull request Apr 26, 2023

Upgrade the toolchain to nightly-2023-04-16 model-checking/kani#2406

Merged

4 tasks

tamird mentioned this pull request Apr 26, 2023

Add new fields for internal test crate Manishearth/compiletest-rs#264

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementing "<test_binary> --list --format json" for use by IDE test explorers / runners #108148

Implementing "<test_binary> --list --format json" for use by IDE test explorers / runners #108148

parthopdas commented Feb 17, 2023 •

edited by Manishearth

Loading

rustbot commented Feb 17, 2023

rustbot commented Feb 17, 2023

parthopdas commented Feb 17, 2023

Manishearth commented Feb 17, 2023

Manishearth commented Feb 17, 2023

parthopdas commented Feb 17, 2023 •

edited

Loading

bjorn3 Feb 17, 2023

parthopdas Feb 17, 2023

bjorn3 Feb 18, 2023

bjorn3 Feb 17, 2023

parthopdas Feb 17, 2023

bjorn3 Feb 17, 2023

parthopdas Feb 17, 2023

Manishearth commented Feb 17, 2023

oli-obk commented Feb 17, 2023

parthopdas commented Feb 17, 2023

parthopdas commented Feb 17, 2023

Manishearth commented Feb 17, 2023

parthopdas commented Feb 17, 2023

parthopdas commented Feb 17, 2023

parthopdas commented Feb 19, 2023

parthopdas commented Mar 3, 2023

oli-obk commented Mar 3, 2023

oli-obk commented Mar 16, 2023

bors commented Mar 16, 2023

Manishearth commented Mar 18, 2023

bors commented Mar 18, 2023

bors commented Mar 18, 2023

rust-log-analyzer commented Mar 18, 2023

ehuss commented Mar 18, 2023

parthopdas commented Mar 18, 2023

bors commented Mar 20, 2023

bors commented Mar 20, 2023

rust-timer commented Mar 20, 2023

	if self.cx.tcx().should_collapse_debuginfo(span) {
	// Walk up the macro expansion chain until we reach a non-expanded span.
	// We also stop at the function body level because no line stepping can occur
	// at the level above that.
	// Use span of the outermost expansion site, while keeping the original lexical scope.
	span = rustc_span::hygiene::walk_chain(span, self.mir.span.ctxt());
	}

Implementing "<test_binary> --list --format json" for use by IDE test explorers / runners #108148

Implementing "<test_binary> --list --format json" for use by IDE test explorers / runners #108148

Conversation

parthopdas commented Feb 17, 2023 • edited by Manishearth Loading

rustbot commented Feb 17, 2023

rustbot commented Feb 17, 2023

parthopdas commented Feb 17, 2023

Manishearth commented Feb 17, 2023

Manishearth commented Feb 17, 2023

parthopdas commented Feb 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Manishearth commented Feb 17, 2023

oli-obk commented Feb 17, 2023

parthopdas commented Feb 17, 2023

parthopdas commented Feb 17, 2023

Manishearth commented Feb 17, 2023

parthopdas commented Feb 17, 2023

parthopdas commented Feb 17, 2023

parthopdas commented Feb 19, 2023

parthopdas commented Mar 3, 2023

oli-obk commented Mar 3, 2023

oli-obk commented Mar 16, 2023

bors commented Mar 16, 2023

Manishearth commented Mar 18, 2023

bors commented Mar 18, 2023

bors commented Mar 18, 2023

rust-log-analyzer commented Mar 18, 2023

ehuss commented Mar 18, 2023

parthopdas commented Mar 18, 2023

bors commented Mar 20, 2023

bors commented Mar 20, 2023

rust-timer commented Mar 20, 2023

Overall result: ❌ regressions - no action needed

parthopdas commented Feb 17, 2023 •

edited by Manishearth

Loading

parthopdas commented Feb 17, 2023 •

edited

Loading