Revive the librarification effort

[Veykril]

Our fork of rustc_abi is quite outdated by now, and unfortunately upstream broke the non nightly build of it (and worse, removed all the generic parameters we depended on) so we can't update it anymore without putting in significant effort into fixing that. This will most likely be a recurring thing so maybe it is time to revive the librarification effort (and potentially crate a working group for that) to lay out a proper plan for such crates. Or at least properly librarify the rustc_abi crate (that is no ugly nightly cfgs)

Obviously not the best place to have this discussion right now, but putting it down here as a reminder (to myself at least).

[HKalbasi]

First thing to do is adding the toggle mentioned in #14846 to switch to the workspace dependencies for rustc crate when r-a is being built inside of the rustc tree, so things like rust-lang/rust#107163 will be detected in the rustc CI. For rustc_lexer, rustc_parse_format, rustc_index, ... it has no problem since rust is backward compatible, and for rustc_abi (which we can't toggle it right now anyway) it means we are in sync with latest nightly instead of latest stable, which is not ideal, but better than the current state where we are several stable release back.

I'm all for librarification, in any shape. It seems like out of tree librarification (like chalk and polonious) made things hard for rustc developers, and in tree librarification (like rustc_lexer) is the way to go, which that toggle can help with it.

About rustc_abi, ideal librarification of it is probably blocked on the librarification of type system, but it doesn't mean that we can't improve the current state.

[Veykril]

With toggle you mean the cfg right?

Having them synced properly is orthogonal to this issue. It's mainly about the crates being usable on a stable context such other consumers (which r-a is a prime example of) can use them.
These would stay in tree I agree, there is not much point in having them be lifted out, likewise we don't need them to have a stable API as we can easily work with API breakage churn. My point being we should make it as easy as possible for the rustc maintainers here as this is a request coming from us and it will make their life a bit more annoying no matter what, so lets keep that to a minimum if possible.