generate MIR for the nightlies' standard library

[oli-obk]

sizes of files in /lib after this PR:

20K	libarena-73fff20938e5b4a0.so
48K	libflate-da0248732d23c024.so
48K	libfmt_macros-aedc6fc60322f45d.so
136K	libgetopts-b7de4ad01093c191.so
64K	libgraphviz-4cddbfded062cc67.so
80K	liblog-244814ed5548eb1d.so
44K	libproc_macro-07e61644036f16f4.so
156K	libproc_macro_plugin-08f1f3a23d0dda62.so
40K	libproc_macro_tokens-cb251390161800b9.so
11M	librustc-989ff71a3ff3f468.so
808K	librustc_back-4035e0e266e96bc8.so
1,4M	librustc_borrowck-9ab1ee1b865353ac.so
780K	librustc_const_eval-b4de6eae684ba589.so
212K	librustc_const_math-ff165bf6aed2dfa9.so
356K	librustc_data_structures-c66020d79d96a10c.so
2,1M	librustc_driver-c54f645e9ae37a10.so
380K	librustc_errors-841d081d1aef6867.so
960K	librustc_incremental-bc29c6e1de862741.so
636K	librustc_lint-8a4590e690bc08ab.so
39M	librustc_llvm-4f865650e23ee0b2.so
1,6M	librustc_metadata-d8c96d0eb7b4f6b3.so
1,6M	librustc_mir-d0713934d969099b.so
496K	librustc_passes-17b84294efc81a54.so
872K	librustc_platform_intrinsics-1626fdb9b5f1c718.so
76K	librustc_plugin-ea8df9fc2fcc4acc.so
204K	librustc_privacy-97dcf115ef96d8e6.so
1,3M	librustc_resolve-cae471b5c573a33b.so
1,5M	librustc_save_analysis-125beb4408fb5450.so
3,6M	librustc_trans-b01bcf4a676e8b18.so
3,7M	librustc_typeck-773fa7021f4cbc93.so
4,8M	librustdoc-cdda68154a149cb4.so
604K	libserialize-134f515a17a30f2c.so
5,0M	libstd-c26b46dac555a4f4.so
1,4M	libsyntax_ext-97f59ffcd48cfaad.so
6,8M	libsyntax-fa53ad3a8b4ce82b.so
124K	libsyntax_pos-ba2dd9baed5c77d7.so
316K	libterm-7beb204b9e7a957c.so
380K	libtest-a8f9282521afb4b8.so
130M	rustlib

sizes of files in /lib before this PR:

20K	libarena-73fff20938e5b4a0.so
48K	libflate-da0248732d23c024.so
36K	libfmt_macros-aedc6fc60322f45d.so
104K	libgetopts-b7de4ad01093c191.so
56K	libgraphviz-4cddbfded062cc67.so
68K	liblog-244814ed5548eb1d.so
44K	libproc_macro-07e61644036f16f4.so
132K	libproc_macro_plugin-08f1f3a23d0dda62.so
36K	libproc_macro_tokens-cb251390161800b9.so
7,9M	librustc-989ff71a3ff3f468.so
548K	librustc_back-4035e0e266e96bc8.so
1,1M	librustc_borrowck-9ab1ee1b865353ac.so
588K	librustc_const_eval-b4de6eae684ba589.so
136K	librustc_const_math-ff165bf6aed2dfa9.so
320K	librustc_data_structures-c66020d79d96a10c.so
2,0M	librustc_driver-c54f645e9ae37a10.so
268K	librustc_errors-841d081d1aef6867.so
760K	librustc_incremental-bc29c6e1de862741.so
520K	librustc_lint-8a4590e690bc08ab.so
39M	librustc_llvm-4f865650e23ee0b2.so
1,3M	librustc_metadata-d8c96d0eb7b4f6b3.so
1,1M	librustc_mir-d0713934d969099b.so
400K	librustc_passes-17b84294efc81a54.so
672K	librustc_platform_intrinsics-1626fdb9b5f1c718.so
60K	librustc_plugin-ea8df9fc2fcc4acc.so
144K	librustc_privacy-97dcf115ef96d8e6.so
976K	librustc_resolve-cae471b5c573a33b.so
1,3M	librustc_save_analysis-125beb4408fb5450.so
2,3M	librustc_trans-b01bcf4a676e8b18.so
2,7M	librustc_typeck-773fa7021f4cbc93.so
3,5M	librustdoc-cdda68154a149cb4.so
532K	libserialize-134f515a17a30f2c.so
4,5M	libstd-c26b46dac555a4f4.so
1,1M	libsyntax_ext-97f59ffcd48cfaad.so
5,3M	libsyntax-fa53ad3a8b4ce82b.so
104K	libsyntax_pos-ba2dd9baed5c77d7.so
260K	libterm-7beb204b9e7a957c.so
336K	libtest-a8f9282521afb4b8.so
116M	rustlib

so there's about a 10% increase in library size

[rust-highfive]

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

[oli-obk]

r? @eddyb

[Mark-Simulacrum]

Does this increase build times? If so, we may want to think about which one we should optimize for. I personally would prefer decreasing build times over decreasing size.

[nagisa]

Why would the end user ever care about having MIR for the libstd?

[bluss]

Sorry for the basic questions, but does this change how programs compile when they use std items? For example: right now, a development build using String::clone links to the release-optimized version shipped in libstd (libcollections).

[oli-obk]

Sorry for the basic questions, but does this change how programs compile when they use std items? For example: right now, a development build using String::clone links to the release-optimized version shipped in libstd (libcollections).

It should not. In #38217 we moved the logic that decided whether to inline code from metadata creation to trans. If anything, that PR has a bug, not this one.

Why would the end user ever care about having MIR for the libstd?

Because they'll be able to interpret arbitrary Rust code through miri.

Does this increase build times? If so, we may want to think about which one we should optimize for. I personally would prefer decreasing build times over decreasing size.

I haven't tested, but to the best of my knowledge metadata encoding takes much less time than compilation

[nagisa]

Because they'll be able to interpret arbitrary Rust code through miri.

I find this argument to not be persuasive, I’d understand if the implementation only unconditionally included MIR in libstd, libcollections and libcore only, not every single crate. The distribution is already pretty big and 10% increase for the whole distribution is not trivial.

I also have my doubts about advertising this particular use-case. Remember that the flag in question is unstable. While it doesn’t prevent anybody from relying on miri, it doesn’t make any behaviour stable either, What this PR does is including MIR into stable libraries unconditionally as well, enabling people to use stable libraries on miri and therefore allowing them to complain once/if we decide we do not actually want to put all the MIR into all the crates.

[oli-obk]

How about only generating it for nightlies?

I can totally see miri not being a relevant use case for this change. I'm fine with closing this and finding another solution. But I at least wanted to try ;)

[oli-obk]

This pr does not unconditionally enable mir for all crates. Downstream everyone still needs to enable the flag. It also doesn't allow miri to suddenly work on stable. But yes, miri can now interpret any crate, stable or not and undoing this change will break that ability. The obvious solution is to use xargo, which will custom build libcore. Not sure what is necessary to make it compile libstd though.

I'll try the xargo route

[nagisa]

I would not object against including full MIR for libstd, libcollections and libcore on nightly only.

[eddyb]

Isn't it dashes, not underscores? 😆

[oli-obk]

nope, this was the correct version (especially since it worked)

[eddyb]

That's... weird? It sounds like a bug, let me check something.

[eddyb]

Ah no we accept both underscores and dashes, but only dashes are canonical (I tried with -C debug-assertions and -C debug_assertions, both appear to work).

[oli-obk]

I changed it so the MIR is only encoded on nightly and dev

[eddyb]

LGTM. CC @rust-lang/compiler @rust-lang/tools

[oli-obk]

This has become obsolete due to https://users.rust-lang.org/t/xargo-v0-3-0-released-build-your-own-std/8571 .

[oli-obk]

I got a little overexcited. xargo 3.0 doesn't work for the host platform.

[nikomatsakis]

Hmm. I also feel the motivation for this is somewhat weak. Although we don't particularly optimize for it, distribution size does matter for some users. It'd be nice if one could "opt in" to it.

Would we ever want this for const fn etc? Seems like no -- or at least, we'd only want it for "reachable" things.

[brson]

If the motivation is to make miri work, there may be other ways. For example xargo could produce a miri std pretty easily.

I don't know that this has a path to stabilization, but distributing special binaries in nightly strongly implies it does. I would not want to do this unless there's clear agreement that this is a thing we want to do on stable. And even then I might suggest we do it another way: by putting mir on all channels but making it unusable without nightly. That seems more consistent with how nightly is currently used - feature exists, can't be used.

[oli-obk]

Ok, I'll investigate xargo more.