compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO. #57514

michaelwoerister · 2019-01-11T13:01:23Z

Some cross-language run-make tests need a Clang compiler that matches the LLVM version of rustc. Since such a compiler usually isn't available these tests (marked with the needs-matching-clang
directive) are ignored by default.

For some CI jobs we do need these tests to run unconditionally though. In order to support this a --force-clang-based-tests flag is added to compiletest. If this flag is specified, compiletest will fail if it can't detect an appropriate version of Clang.

@rust-lang/infra The PR doesn't yet enable the tests yet. Do you have any recommendation for which jobs to enable them?

cc #57438

r? @alexcrichton

rust-highfive · 2019-01-11T13:40:52Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:0a6a7b73:start=1547211744077826380,finish=1547211813523019759,duration=69445193379
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-6.0
---

[00:03:20] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:03:20] tidy error: /checkout/src/bootstrap/test.rs:1109: line longer than 100 chars
[00:03:22] some tidy checks failed
[00:03:22] 
[00:03:22] 
[00:03:22] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:22] 
[00:03:22] 
[00:03:22] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:22] Build completed unsuccessfully in 0:00:46
[00:03:22] Build completed unsuccessfully in 0:00:46
[00:03:22] make: *** [tidy] Error 1
[00:03:22] Makefile:69: recipe for target 'tidy' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:17e14d0e
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Fri Jan 11 13:07:04 UTC 2019
---
travis_time:end:11d9d38a:start=1547212024644878298,finish=1547212024649309522,duration=4431224
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:11d00126
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:2eccbd58
travis_time:start:2eccbd58
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:07f89efa
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

alexcrichton · 2019-01-11T17:46:28Z

src/bootstrap/test.rs

+
+        if let Some(var) = env::var_os("RUSTBUILD_FORCE_CLANG_BASED_TESTS") {
+            match &var.to_string_lossy()[..] {
+                "1" | "yes" | "on" => {


Heh since this is an internal env var I think matching on only "yes" or "1" is probably fine, we tend to not really need that much error handling and support for internal features like this

I wanted to try and make it less likely to have a typo causing things to be silently ignored. What would be really needed though would be fuzzy matching on env var names...

True yeah, although we could also just check for existence here instead of checking for value perhaps?

That's what wanted to do at first but then I remembered the CARGO_INCREMENTAL=0 confusion. Since it's not really complicated to support a few different options here, I just did that.

src/tools/compiletest/src/main.rs

src/test/run-make-fulldeps/cross-lang-lto-clang/Makefile

michaelwoerister · 2019-01-15T17:11:12Z

I've simplified how Clang-based tests are set up. There's just a --run-clang-based-tests-with option for compiletest. If the option is supplied, clang-tests are run with the given clang executable. If it is not supplied, the tests are ignored.

Which builders should we enable the tests on?

alexcrichton · 2019-01-15T18:34:22Z

Currently I believe we only enable LLDB on the dist builders for OSX, but we'll probably want to pick a Linux builder which isn't taking too long today and run it there. I don't think it particularly matters which one we choose!

michaelwoerister · 2019-01-16T17:05:43Z

Oh no, one run-make-fulldeps test ignored, but it doesn't say which one :(

alexcrichton · 2019-01-17T00:59:13Z

src/ci/docker/x86_64-gnu/Dockerfile

@@ -16,9 +16,13 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 COPY scripts/sccache.sh /scripts/
 RUN sh /scripts/sccache.sh

+ENV RUSTBUILD_FORCE_CLANG_BASED_TESTS yes


Checking a recent test run this builder is already running too long at 2h11m, perhaps we can pick a faster builder like x86_64-gnu-debug to run these tests in?

But that doesn't seem to run tests?

alexcrichton · 2019-01-17T00:59:33Z

Was that when testing locally? Or when testing this on CI? (the Travis run for this PR doesn't include the builder activated here)

Some cross-language run-make tests need a Clang compiler that matches the LLVM version of rustc. Since such a compiler usually isn't available these tests (marked with the "needs-matching-clang" directive) are ignored by default. For some CI jobs we do need these tests to run unconditionally though. In order to support this a --force-clang-based-tests flag is added to compiletest. If this flag is specified, compiletest will fail if it can't detect an appropriate version of Clang.

michaelwoerister · 2019-01-17T15:53:11Z

Was that when testing locally? Or when testing this on CI? (the Travis run for this PR doesn't include the builder activated here)

Oh right, the pre-check is using the LLVM-6.0 image.

michaelwoerister · 2019-01-17T16:27:25Z

I switched to the x86_64-debug builder and made it run the clang-based tests (and only those):
6e0ffb3#diff-f06eb47b50daaf84c58fe00653913115R30

Is that what you meant, @alexcrichton?

alexcrichton · 2019-01-18T22:22:45Z

Ah true that wasn't running any tests... That looks fine, I really don't have an opinion about where this should go other than that this is quite a large ask for CI to build all of clang and run more tests. We need to be careful to not regress cycle time which is already really really bad. If possible can this run through @bors: try to get ballpark estimates on timing?

rust-highfive · 2019-01-22T15:03:51Z

The job x86_64-gnu-debug of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:05b93990:start=1548167229922750386,finish=1548167301305231579,duration=71382481193
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-debug
---
[00:01:10] Step 7/8 : ENV RUST_CONFIGURE_ARGS       --build=x86_64-unknown-linux-gnu       --enable-debug       --enable-lld       --enable-lldb       --enable-optimize
[00:01:10]  ---> Running in 8f73d3d91389
[00:01:10] Removing intermediate container 8f73d3d91389
[00:01:10]  ---> cc2ddca93b9d
[00:01:10] Step 8/8 : ENV SCRIPT   python2.7 ../x.py build &&   python2.7 ../x.py test src/test/run-make-fulldeps --test-args clang
[00:01:10] Removing intermediate container a9c725490877
[00:01:10]  ---> bf67eb77a819
[00:01:10] Successfully built bf67eb77a819
[00:01:10] Successfully tagged rust-ci:latest
[00:01:10] Successfully tagged rust-ci:latest
[00:01:10] Built container sha256:bf67eb77a819908ce3d199751d776becb041807bbc6198b9c2ec5351c31c7937
[00:01:10] Uploading finished image to s3://rust-lang-ci-sccache2/docker/443850f43276f1b1220ffd0cf910f2f5a59531a55b07fe397c725d708ad26207600e68981d9be3f02cc20d364dbf6404db559718f4ee2b09acee812129e0468e
[00:01:42] upload failed: - to s3://rust-lang-ci-sccache2/docker/443850f43276f1b1220ffd0cf910f2f5a59531a55b07fe397c725d708ad26207600e68981d9be3f02cc20d364dbf6404db559718f4ee2b09acee812129e0468e Unable to locate credentials

[00:01:42] travis_time:end:07ac3814:start=1548167321659331995,finish=1548167412685253398,duration=91025921403
[CI_JOB_NAME=x86_64-gnu-debug]
[00:01:43] [CI_JOB_NAME=x86_64-gnu-debug]
---
[00:34:40] -- Looking for sys/resource.h - found
[00:34:40] -- Clang version: 8.0.0
[00:34:40] -- Performing Test CXX_SUPPORTS_NO_NESTED_ANON_TYPES_FLAG
[00:34:40] -- Performing Test CXX_SUPPORTS_NO_NESTED_ANON_TYPES_FLAG - Failed
[00:34:40] -- Could NOT find Z3 (missing:  Z3_LIBRARIES Z3_INCLUDE_DIR) (Required is exact version "4.7.1")
[00:34:41] CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
[00:34:41]   Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
[00:34:41] Call Stack (most recent call first):
[00:34:41]   /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:388 (_FPHSA_FAILURE_MESSAGE)
[00:34:41]   /usr/share/cmake-3.5/Modules/FindPythonLibs.cmake:265 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
[00:34:41]   /checkout/src/tools/lldb/cmake/modules/LLDBConfig.cmake:181 (find_package)
[00:34:41] 
[00:34:41] 
[00:34:41] -- Configuring incomplete, errors occurred!
[00:34:41] -- Configuring incomplete, errors occurred!
[00:34:41] See also "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/CMakeFiles/CMakeOutput.log".
[00:34:41] See also "/checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/CMakeFiles/CMakeError.log".
[00:34:41] command did not execute successfully, got: exit code: 1
[00:34:41] 
[00:34:41] 
[00:34:41] build script failed, must exit now', /cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.33/src/lib.rs:773:5
[00:34:41]  finished in 10.831
[00:34:41] travis_fold:end:llvm

[00:34:41] travis_time:end:llvm:start=1548169380711736177,finish=1548169391543474609,duration=10831738432
---
travis_time:end:04b08978:start=1548169392846867960,finish=1548169392853672676,duration=6804716
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0c07d7d8
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0bfb976c
travis_time:start:0bfb976c
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:1af8b019
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

michaelwoerister · 2019-01-22T15:27:26Z

Huh, don't know what that is about. Cmake can't find python?

alexcrichton · 2019-01-22T22:19:39Z

Oh the containers all have varying amounts of software on them, you may need to update one aspect here or there to install more things like python. Also FWIW PR builds won't be great for timing because they don't have access to sccache credentials which means they can't actually write any contents to the cache, only pull them

michaelwoerister · 2019-01-24T12:21:31Z

@bors try

bors · 2019-01-24T12:21:42Z

⌛ Trying commit ca6053e with merge a315d17...

@alexcrichton

compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO. Some cross-language run-make tests need a Clang compiler that matches the LLVM version of `rustc`. Since such a compiler usually isn't available these tests (marked with the `needs-matching-clang` directive) are ignored by default. For some CI jobs we do need these tests to run unconditionally though. In order to support this a `--force-clang-based-tests` flag is added to compiletest. If this flag is specified, `compiletest` will fail if it can't detect an appropriate version of Clang. @rust-lang/infra The PR doesn't yet enable the tests yet. Do you have any recommendation for which jobs to enable them? cc #57438 r? @alexcrichton

michaelwoerister · 2019-01-30T14:39:12Z

So LLD doesn't seem to make a huge difference but it's still nice to have it available.
Anyway, the PR is pretty much done now.

bors · 2019-01-30T14:52:31Z

☀️ Test successful - checks-travis
State: approved= try=True

alexcrichton · 2019-01-30T15:07:22Z

Ok! Want to back out the changes to .travis.yml? Otherwise r=me

michaelwoerister · 2019-01-30T15:25:16Z

Ah, good catch! Done.

@bors r=alexcrichton

bors · 2019-01-30T15:25:17Z

📌 Commit b17c10d has been approved by alexcrichton

bors · 2019-01-30T18:55:44Z

⌛ Testing commit b17c10d with merge dce0744fc43e1ce55ab841a40940b5afbe030909...

bors · 2019-01-30T21:04:02Z

💔 Test failed - status-appveyor

michaelwoerister · 2019-01-31T11:05:43Z

@bors retry

bors · 2019-01-31T11:07:48Z

⌛ Testing commit b17c10d with merge d30b99f...

@alexcrichton

compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO. Some cross-language run-make tests need a Clang compiler that matches the LLVM version of `rustc`. Since such a compiler usually isn't available these tests (marked with the `needs-matching-clang` directive) are ignored by default. For some CI jobs we do need these tests to run unconditionally though. In order to support this a `--force-clang-based-tests` flag is added to compiletest. If this flag is specified, `compiletest` will fail if it can't detect an appropriate version of Clang. @rust-lang/infra The PR doesn't yet enable the tests yet. Do you have any recommendation for which jobs to enable them? cc #57438 r? @alexcrichton

bors · 2019-01-31T13:40:06Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: alexcrichton
Pushing d30b99f to master...

rust-highfive assigned alexcrichton Jan 11, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jan 11, 2019

michaelwoerister force-pushed the xlto-tests branch from 6491bc8 to 8fc4d91 Compare January 11, 2019 14:16

alexcrichton reviewed Jan 11, 2019

View reviewed changes

tesuji reviewed Jan 11, 2019

View reviewed changes

src/test/run-make-fulldeps/cross-lang-lto-clang/Makefile Outdated Show resolved Hide resolved

michaelwoerister force-pushed the xlto-tests branch 2 times, most recently from fc95038 to 4ad6205 Compare January 15, 2019 17:08

alexcrichton reviewed Jan 17, 2019

View reviewed changes

michaelwoerister added 4 commits January 17, 2019 16:49

Add an end-to-end run-make test for cross-lang LTO.

75dcf9e

Support clang-based run-make tests in rustbuild.

ea4fb95

compiletest: Simplify handling of Clang-based tests.

b38125c

michaelwoerister force-pushed the xlto-tests branch from f4ef08a to 6e0ffb3 Compare January 17, 2019 16:24

michaelwoerister force-pushed the xlto-tests branch from aec74bf to 821e715 Compare January 22, 2019 14:18

Enable Clang-based tests on x86_64-gnu-debug builder.

48cb04f

michaelwoerister force-pushed the xlto-tests branch from 821e715 to ca6053e Compare January 24, 2019 12:21

michaelwoerister added 5 commits January 30, 2019 16:21

Add missing packages.

4460e7c

bootstrap: Make LLD available to run-make tests.

9c29517

ci: Use clang as the C++ compiler for x86_64-gnu-debug.

4f534ff

bootstrap: Expose LLVM_USE_LINKER cmake option to config.toml.

dc20c8c

CI: Use lld for linking LLVM in the x86_64-gnu-debug image.

b17c10d

michaelwoerister force-pushed the xlto-tests branch from ccbab3f to b17c10d Compare January 30, 2019 15:22

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 30, 2019

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 30, 2019

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 31, 2019

mati865 mentioned this pull request Jan 31, 2019

proc_macro: make TokenStream::from_streams pre-allocate its vector. #58001

Merged

bors merged commit b17c10d into rust-lang:master Jan 31, 2019

pnkfelix mentioned this pull request Jan 31, 2019

AppVeyor spuriously killing a dist- job with message "Build exceeded allowed resource quotas" #58025

Closed

michaelwoerister mentioned this pull request Feb 1, 2019

Properly test cross-language LTO #57438

Open

mati865 mentioned this pull request Feb 6, 2019

Linking with LLD #39915

Open

Zalathar mentioned this pull request Jun 8, 2024

Remove empty test suite tests/run-make-fulldeps #126155

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO. #57514

compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO. #57514

michaelwoerister commented Jan 11, 2019

rust-highfive commented Jan 11, 2019

alexcrichton Jan 11, 2019

michaelwoerister Jan 14, 2019

alexcrichton Jan 14, 2019

michaelwoerister Jan 15, 2019

michaelwoerister commented Jan 15, 2019

alexcrichton commented Jan 15, 2019

michaelwoerister commented Jan 16, 2019

alexcrichton Jan 17, 2019

michaelwoerister Jan 17, 2019

alexcrichton commented Jan 17, 2019

michaelwoerister commented Jan 17, 2019

michaelwoerister commented Jan 17, 2019

alexcrichton commented Jan 18, 2019

rust-highfive commented Jan 22, 2019

michaelwoerister commented Jan 22, 2019

alexcrichton commented Jan 22, 2019

michaelwoerister commented Jan 24, 2019

bors commented Jan 24, 2019

michaelwoerister commented Jan 30, 2019

bors commented Jan 30, 2019

alexcrichton commented Jan 30, 2019

michaelwoerister commented Jan 30, 2019

bors commented Jan 30, 2019

bors commented Jan 30, 2019

bors commented Jan 30, 2019

michaelwoerister commented Jan 31, 2019

bors commented Jan 31, 2019

bors commented Jan 31, 2019

compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO. #57514

compiletest: Support opt-in Clang-based run-make tests and use them for testing xLTO. #57514

Conversation

michaelwoerister commented Jan 11, 2019

rust-highfive commented Jan 11, 2019

alexcrichton Jan 11, 2019

Choose a reason for hiding this comment

michaelwoerister Jan 14, 2019

Choose a reason for hiding this comment

alexcrichton Jan 14, 2019

Choose a reason for hiding this comment

michaelwoerister Jan 15, 2019

Choose a reason for hiding this comment

michaelwoerister commented Jan 15, 2019

alexcrichton commented Jan 15, 2019

michaelwoerister commented Jan 16, 2019

alexcrichton Jan 17, 2019

Choose a reason for hiding this comment

michaelwoerister Jan 17, 2019

Choose a reason for hiding this comment

alexcrichton commented Jan 17, 2019

michaelwoerister commented Jan 17, 2019

michaelwoerister commented Jan 17, 2019

alexcrichton commented Jan 18, 2019

rust-highfive commented Jan 22, 2019

michaelwoerister commented Jan 22, 2019

alexcrichton commented Jan 22, 2019

michaelwoerister commented Jan 24, 2019

bors commented Jan 24, 2019

michaelwoerister commented Jan 30, 2019

bors commented Jan 30, 2019

alexcrichton commented Jan 30, 2019

michaelwoerister commented Jan 30, 2019

bors commented Jan 30, 2019

bors commented Jan 30, 2019

bors commented Jan 30, 2019

michaelwoerister commented Jan 31, 2019

bors commented Jan 31, 2019

bors commented Jan 31, 2019