Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spurious timeout in dist builders #48827

Closed
alexcrichton opened this issue Mar 7, 2018 · 4 comments
Closed

Spurious timeout in dist builders #48827

alexcrichton opened this issue Mar 7, 2018 · 4 comments
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@alexcrichton
Copy link
Member

This is a reopening of #48192 now that the sccache issue identified in that issue has been fixed. The dist builders are still occasionally timing out over three hours, and we need to get that under control!

In a discussion today some various possibilities arose, but unfortunately there doesn't seem to be many silver bullets here:

  • A long-term change could be to perhaps leverage build stages on Travis. Build stages would then allow us to avoid timeouts almost entirely and perhaps even shorten build times having certain sequences of the build be much more parallel than before, leveraging available capacity on Travis. This is of course a pretty significant investment, though, and may be difficult to implement.
  • Skip documenting cross-compiled builds altogether. The ARM documentation is probably really similar to the x86 documentation, and skipping docs entirely would skip rustdoc for the host, rustbook, and building the docs themselves. More information is needed though about how much time this would save.
  • Precompiling possible tools like rustfmt and Cargo. This doesn't work for tools that depend on rustc/unstable features, but it's possible that we could cross-compile binaries in their own CI systems. The downside of this though is that it spreads out already complicated CI configuration, and the RLS itself depends on both rustfmt/cargo in library form so we may have to build them anyway...

I'll try to keep this updated with known causes of timeouts and possible mitigation strategies.
@alexcrichton alexcrichton added A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC labels Mar 7, 2018
@alexcrichton
Copy link
Member Author

cc @rust-lang/infra

@alexcrichton
Copy link
Member Author

Disabling docs locally for me shaved only 5 minutes off the build, going from 1h10m to 1h5m

@kennytm kennytm mentioned this issue Mar 8, 2018
Open
5 tasks
@alexcrichton alexcrichton mentioned this issue Mar 19, 2018
Merged
alexcrichton added a commit to alexcrichton/rust that referenced this issue Mar 26, 2018
@alexcrichton
rustbuild: Disable docs on cross-compiled builds
bd28641 
This commit disables building documentation on cross-compiled compilers, for
example ARM/MIPS/PowerPC/etc. Currently I believe we're not getting much use out
of these documentation artifacts and they often take 10-15 minutes total to
build as it requires building rustdoc/rustbook and then also generating all the
documentation, especially for the reference and the book itself.

In an effort to cut down on the amount of work that we're doing on dist CI
builders in light of recent timeouts this was some relatively low hanging fruit
to cut which in theory won't have much impact on the ecosystem in the hopes that
the documentation isn't used too heavily anyway.

While initial analysis in rust-lang#48827 showed only shaving 5 minutes off local builds
the same 5 minute conclusion was drawn from rust-lang#48826 which ended up having nearly
a half-hour impact on the bots. In that sense I'm hoping that we can land this
and test out what happens on CI to see how it affects timing.

Note that all tier 1 platforms, Windows, Mac, and Linux, will continue to
generate documentation.
kennytm added a commit to kennytm/rust that referenced this issue Mar 26, 2018
@kennytm
Rollup merge of rust-lang#49187 - alexcrichton:no-cross-docs, r=kennytm
ede4cd9 
rustbuild: Disable docs on cross-compiled builds

This commit disables building documentation on cross-compiled compilers, for
example ARM/MIPS/PowerPC/etc. Currently I believe we're not getting much use out
of these documentation artifacts and they often take 10-15 minutes total to
build as it requires building rustdoc/rustbook and then also generating all the
documentation, especially for the reference and the book itself.

In an effort to cut down on the amount of work that we're doing on dist CI
builders in light of recent timeouts this was some relatively low hanging fruit
to cut which in theory won't have much impact on the ecosystem in the hopes that
the documentation isn't used too heavily anyway.

While initial analysis in rust-lang#48827 showed only shaving 5 minutes off local builds
the same 5 minute conclusion was drawn from rust-lang#48826 which ended up having nearly
a half-hour impact on the bots. In that sense I'm hoping that we can land this
and test out what happens on CI to see how it affects timing.

Note that all tier 1 platforms, Windows, Mac, and Linux, will continue to
generate documentation.
TimNN added a commit to TimNN/rust that referenced this issue Mar 26, 2018
@TimNN
Rollup merge of rust-lang#49187 - alexcrichton:no-cross-docs, r=kennytm
bd3db2b 
rustbuild: Disable docs on cross-compiled builds

This commit disables building documentation on cross-compiled compilers, for
example ARM/MIPS/PowerPC/etc. Currently I believe we're not getting much use out
of these documentation artifacts and they often take 10-15 minutes total to
build as it requires building rustdoc/rustbook and then also generating all the
documentation, especially for the reference and the book itself.

In an effort to cut down on the amount of work that we're doing on dist CI
builders in light of recent timeouts this was some relatively low hanging fruit
to cut which in theory won't have much impact on the ecosystem in the hopes that
the documentation isn't used too heavily anyway.

While initial analysis in rust-lang#48827 showed only shaving 5 minutes off local builds
the same 5 minute conclusion was drawn from rust-lang#48826 which ended up having nearly
a half-hour impact on the bots. In that sense I'm hoping that we can land this
and test out what happens on CI to see how it affects timing.

Note that all tier 1 platforms, Windows, Mac, and Linux, will continue to
generate documentation.
@crlf0710
Copy link
Member

@kennytm Maybe this is also outdated after the migration to azure pipelines.

@alexcrichton
Copy link
Member Author

Hasn't happened in quite some time so closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alexcrichton @crlf0710