Make json test output formatter represent "test_count" as num #53867

cwndrws · 2018-08-31T19:48:02Z

rust-highfive · 2018-08-31T19:48:12Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @sfackler (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

sfackler · 2018-08-31T19:49:35Z

What downstream code consumes this format? Is this a backwards compatibility concern?

cwndrws · 2018-08-31T19:52:53Z

@sfackler I'm working on a tool that consumes this. It's not backwards compatibility concern. It just seems wrong.

rust-highfive · 2018-08-31T21:24:38Z

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:52:46] ....................................................................................................
[00:52:50] ....................................................................................................
[00:52:53] ...............i....................................................................................
[00:52:56] ....................................................................................................
[00:52:59] ................................................................iiiiiiiii...........................
[00:53:04] ....................................................................................................
[00:53:08] ....................................................................................................
[00:53:11] ............................................i.......................................................
[00:53:14] ..............................................................................................i.i..i
---
travis_time:start:test_run-make-fulldeps
Check compiletest suite=run-make-fulldeps mode=run-make (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:22:52] 
[01:22:52] running 190 tests
[01:23:23] .......................................................................................F............
[01:24:17] .........................................................................................test [run-make] run-make-fulldeps/long-linker-command-lines has been running for over 60 seconds
[01:24:58] thread 'main' panicked at 'Some tests failed', tools/compiletest/src/main.rs:497:22
[01:24:58] failures:
[01:24:58] 
[01:24:58] ---- [run-make] run-make-fulldeps/libtest-json stdout ----
[01:24:58] ---- [run-make] run-make-fulldeps/libtest-json stdout ----
[01:24:58] 
[01:24:58] error: make failed
[01:24:58] status: exit code: 2
[01:24:58] command: "make"
[01:24:58] stdout:
[01:24:58] ------------------------------------------
[01:24:58] make[1]: Entering directory '/checkout/src/test/run-make-fulldeps/libtest-json'
[01:24:58] LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib:" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json  --test f.rs
[01:24:58] RUST_BACKTRACE=0 LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib:" /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json/f -Z unstable-options --test-threads=1 --format=json > /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json/libtest-json-output.json || true
[01:24:58] cat /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json/libtest-json-output.json | "/usr/bin/python2.7" validate_json.py
[01:24:58] # Compare to output file
[01:24:58] diff output.json /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json/libtest-json-output.json
[01:24:58] 1c1
[01:24:58] < { "type": "suite", "event": "started", "test_count": "4" }
[01:24:58] ---
[01:24:58] > { "type": "suite", "event": "started", "test_count": 4 }
[01:24:58] 10c10
[01:24:58] < { "type": "suite", "event": "failed", "passed": 2, "failed": 1, "allowed_fail": 0, "ignored": 1, "measured": 0, "filtered_out": "0" }
[01:24:58] ---
[01:24:58] > { "type": "suite", "event": "failed", "passed": 2, "failed": 1, "allowed_fail": 0, "ignored": 1, "measured": 0, "filtered_out": 0 }
[01:24:58] Makefile:8: recipe for target 'all' failed
[01:24:58] make[1]: Leaving directory '/checkout/src/test/run-make-fulldeps/libtest-json'
[01:24:58] ------------------------------------------
[01:24:58] stderr:
[01:24:58] ------------------------------------------
[01:24:58] ------------------------------------------
[01:24:58] make[1]: *** [all] Error 1
[01:24:58] ------------------------------------------
[01:24:58] 
[01:24:58] 
[01:24:58] thread '[run-make] run-make-fulldeps/libtest-json' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3196:9
[01:24:58] 
[01:24:58] 
[01:24:58] failures:
[01:24:58]     [run-make] run-make-fulldeps/libtest-json
[01:24:58]     [run-make] run-make-fulldeps/libtest-json
[01:24:58] 
[01:24:58] test result: FAILED. 189 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:24:58] 
[01:24:58] 
[01:24:58] 
[01:24:58] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/run-make-fulldeps" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-make" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-5.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "5.0.0\n" "--system-llvm" "--cc" "cc" "--cxx" "c++" "--cflags" "-ffunction-sections -fdata-sections -fPIC -m64" "--llvm-components" "aarch64 aarch64asmparser aarch64asmprinter aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils all all-targets amdgpu amdgpuasmparser amdgpuasmprinter amdgpucodefPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -ffunction-sections -fdata-sections -O2 -DNDEBUG -g1  -fno-exceptions -DLLVM_BUILD_GLOBAL_ISEL -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS" "--ar" "ar" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:24:58] 
[01:24:58] 
[01:24:58] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:24:58] Build completed unsuccessfully in 0:36:27
[01:24:58] Build completed unsuccessfully in 0:36:27
[01:24:58] make: *** [check] Error 1
[01:24:58] Makefile:58: recipe for target 'check' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:2f6a4e72
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
---
travis_time:end:11050c4c:start=1535750674676625774,finish=1535750674751027249,duration=74401475
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:09f2804d
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:07d7bc78
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

zackmdavis · 2018-09-02T01:14:10Z

@sfackler

Is this a backwards compatibility concern?

This JSON output format for tests (note, this is different from --error-format json for diagnostics; I was confused at first) seems to have been introduced in December after the dev-tools team decided that no RFC was needed to land it. That sounds like we haven't made any stability promises yet.

sfackler · 2018-09-02T20:39:15Z

r? @rust-lang/dev-tools

nrc · 2018-09-02T23:09:46Z

cc @djrenren thoughts on this?

djrenren · 2018-09-02T23:19:12Z

Hmm looks like the right call but I'm worried about tool breakage. Most people don't have the time to worry about stability promises. If it works in stable we should generally support it.

nrc · 2018-09-03T01:27:38Z

I don't think anyone is using the JSON test output. cc @vlad20012 - does IntelliJ use it?

djrenren · 2018-09-04T17:58:27Z

Hmm looked through the intellij rust repo and didn't see any references to it. Could be totally fine.

vlad20012 · 2018-09-04T21:25:50Z

does IntelliJ use it?

Not really. We have a PR, but it is not merged yet. So, it's not a problem for us.

nrc · 2018-09-04T21:28:48Z

@bors: r+

bors · 2018-09-04T21:28:49Z

📌 Commit 1c3dc9a has been approved by nrc

bors · 2018-09-05T09:58:03Z

⌛ Testing commit 1c3dc9a with merge b0297f3...

… r=nrc Make json test output formatter represent "test_count" as num fixes #53866

bors · 2018-09-05T12:20:10Z

☀️ Test successful - status-appveyor, status-travis
Approved by: nrc
Pushing b0297f3 to master...

Make json test output formatter represent "test_count" as num

347c6b7

rust-highfive assigned sfackler Aug 31, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 31, 2018

Make json test output formatter represent "filtered_out" as num

b9de64a

Fix tests for json formatting

1c3dc9a

rust-highfive unassigned sfackler Sep 2, 2018

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 4, 2018

bors added a commit that referenced this pull request Sep 5, 2018

Auto merge of #53867 - cwndrws:json-test-formatter-test-count-as-num,…

b0297f3

… r=nrc Make json test output formatter represent "test_count" as num fixes #53866

bors merged commit 1c3dc9a into rust-lang:master Sep 5, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make json test output formatter represent "test_count" as num #53867

Make json test output formatter represent "test_count" as num #53867

cwndrws commented Aug 31, 2018

rust-highfive commented Aug 31, 2018

sfackler commented Aug 31, 2018

cwndrws commented Aug 31, 2018

rust-highfive commented Aug 31, 2018

zackmdavis commented Sep 2, 2018

sfackler commented Sep 2, 2018

nrc commented Sep 2, 2018

djrenren commented Sep 2, 2018

nrc commented Sep 3, 2018

djrenren commented Sep 4, 2018

vlad20012 commented Sep 4, 2018

nrc commented Sep 4, 2018

bors commented Sep 4, 2018

bors commented Sep 5, 2018

bors commented Sep 5, 2018

Make json test output formatter represent "test_count" as num #53867

Make json test output formatter represent "test_count" as num #53867

Conversation

cwndrws commented Aug 31, 2018

rust-highfive commented Aug 31, 2018

sfackler commented Aug 31, 2018

cwndrws commented Aug 31, 2018

rust-highfive commented Aug 31, 2018

zackmdavis commented Sep 2, 2018

sfackler commented Sep 2, 2018

nrc commented Sep 2, 2018

djrenren commented Sep 2, 2018

nrc commented Sep 3, 2018

djrenren commented Sep 4, 2018

vlad20012 commented Sep 4, 2018

nrc commented Sep 4, 2018

bors commented Sep 4, 2018

bors commented Sep 5, 2018

bors commented Sep 5, 2018