V2

[edunham]

Invokes https://github.com/edunham/v2_rust_metadata/ to build channel--.toml. It'll look like https://gist.github.com/edunham/d21384cdce5824bd603c when the latest version gets run (instead of the old version that ran last nightly, because I accidentally had a commit instead of branch name pinned in the installation line in requirements.txt)

[rust-highfive]

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

[brson]

I want to test this live with rustup before merging and deployment.

[alexcrichton]

lgtm, makes me want even more to get the buildmaster under docker or ansible or something like that to automate these changes to install dependencies and whatnot.

[brson]

rustup.sh can install from the v2 manifests in dev now, but there are some issues with the output still that prevent it from installing std. Some of these were in the pasted example, and some only in the live version. I should have looked at the example more closely earlier.

https://dev-static.rust-lang.org/dist/2015-11-16/channel-rust-nightly.toml

etherpad

No rust-std package. The section for the rust-std package does not exist. It looks correct in your example (except for the missing .target which I think is fixed).

Extra rust-std-$triple package. I didn't notice this was in the example you shared too. This is probably a result of interpreting a components file to make the package list. Looks like

[pkg.rust-std-x86_64-pc-windows-msvc]
    available = true
    version = "1.6.0-nightly (00aa3cbec 2015-11-09)"

Incorrect rust-std package name in rust component list. Proboably related to above issues with rust-std. Should be rust-std, not rust-std-$triple. Example.

Missing targets for non-rust packages. cargo, rust-docs, and rustc don't have any target information or urls to their installers. Only in dev. Example looks fine.

Missing mingw packages. Only in dev.

No extensions listed. This only happens in dev.

Incorrect extensions. The only packages that should be extensions are rust-std. In the example it's only rust-docs that are extensions. Example.

Missing extensions. The live version of the manifest lists no extensions.

Available flag is at the wrong level. This flag needs to indicate which tarballs don't exist, but today the available flag is at the package level not the package.target level. In dev I'd expect to see the installers for e.g. darwin to be available = false.

[brson]

To make this all work reliably you may need to hardcode exactly what packages for which targets you expect to see, though maybe that can be derived from information rust-buildbot has.

It confusing and sad that in rust-buildbot 'component' means yet another thing from in rust-installer.

[edunham]

I fixed each of @brson's specific points from #44 (comment) yesterday, however we figured out that the manifest builder also needs to search back through previous dates until it finds a Cargo if there wasn't a Cargo built that night. Working on that this morning.

There won't be any additional commits necessary on this PR to make that change; it's all going into https://github.com/edunham/v2_rust_metadata/ right now which is then installed to the virtualenv. I'll comment again here once I have it examining the cargo-dist/* folders to find the most recent cargo.

[edunham]

I talked with Alex last week about how to correctly structure this; current PR will actually run the script at the wrong time.

Correct solution:

• Add step to finish_dist to write current date into a cargo_date file only when building Cargo, after the manifest generation.
• Run manifest generation after packaging, not during every finish_dist. (around https://github.com/rust-lang/rust-buildbot/blob/master/master/master.cfg#L1459)
• Possibly stop deleting the rustc dist directory when done uploading rustc artifacts in packaging builds, since it just gets re-downloaded around https://github.com/rust-lang/rust-buildbot/blob/master/master/master.cfg#L1380

[aneeshusa]

@alexcrichton If you're still looking to get the buildmaster under CM, I've been helping the Servo team with the Salt configuration they use to coordinate their Buildbot setup. I'd be happy to help create a similar Salt configuration for this repo if you'd like.

[brson]

Closing in favor of #66.