External tracking: multiple targets in single cargo process

[khyperia]

For a while, we've been talking about trying to take our absolutely horrifying, fragile, confusing hacks that let users embed rust-gpu programs in host programs and putting them into cargo itself. For example, this discussion (unfortunately in a private section of our discord).

The high-level goal is to basically remove spirv-builder and make its functionality part of cargo itself.

The extremely high level view of how it works, right now, is: "The user writes x86 crate, with a build.rs. There's a build-dependency on spirv-builder, which itself depends on rustc_codegen_spirv. Spirv-builder then invokes a nested cargo process (within the user's build.rs), passing in rustc_codegen_spirv as the backend to use, and using --target spirv-xyz. It then takes the resulting binary and spews it out of the user's build.rs to make it embeddable in the host x86 crate". We've spent countless, countless hours finding an extremely narrow, fragile path that makes it just barely work on most days, but when thursdays roll around, Thor sneezes and an undecipherable error from halfway around the world comes shooting out.

My theory of how this would be integrated into cargo (only a theory, could be implemented in some other way) is to make cargo aware of building multiple targets in a single cargo process invocation. That means that an x86 crate could declare a dependency on a SPIR-V crate (or a wasm crate, too!), and that dependency would get built, binary blob result gathered, and embedded into the host x86 crate. How those dependencies are specified and configured, how it's embedded, how everything works, are very very open questions.

This issue is for the high-level tracking of investigating what design work should be done, and how to go about implementing it - mainly so that this idea doesn't get lost in the depths of chat logs (again). I don't expect detailed design docs to be in this thread or whatever, I mostly intend this as a sticky note of "do this!!" ✨

Key Cargo issues tracking the development of new functionality to support this:

• Tracking Issue for RFC 3028: Allow "artifact dependencies" on bin, cdylib, and staticlib crates rust-lang/cargo#9096
• Tracking Issue for multidep rust-lang/cargo#10030
• Tracking issues for the -Zmultitarget feature rust-lang/cargo#8176 - probably not needed
• Implement "artifact dependencies" (RFC-3028) rust-lang/cargo#9992
• Implement 'multidep' (RFC-3176) rust-lang/cargo#10061

[repi]

Thanks for filing and detailing! Great to have this as a tracking and coordination issue for this work. Can discuss in posts here (or linked separate issues) while keeping your top post up to date with links and current state of this as we find out (and hopefully) progress more towards this.

Have been a few related RFCs and partial support in Cargo for building crates for multiple targets and dependencies across targets. Added a few links in the top post. Don't know enough about them to know how far they get vs what we need here though but that feels like a good start to investigate also for someone that knows more on the Cargo side (and for us to describe and show the use case here in rust-gpu more).

On another related note, we want this type of "Cargo multi target support" for WASM in a quite similar way also; to be able to have a crate built for native that depends on another crate which is built for wasm32-unknown-unknown and be able to build all of them with a single cargo build. Today for our WASM modules we have them in a separate workspace and manually build both instead. That said, the WASM use case is not as important as the SPIRV / rust-gpu use case we have here, but thought I should mention it.

Another related note and Cargo feature we've been wanting to have that is a minimal building block for this also is to be able to specify which targets a crate can be built for (also wanted both for shaders and for wasm):

• Be able to specify which targets are valid in a workspace rust-ecosystem#41
• Allow specifying a set of supported target platforms in Cargo.toml rust-lang/cargo#6179