Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Libgccjit codegen #87260

Merged
merged 26 commits into from
Sep 29, 2021
Merged

Libgccjit codegen #87260

merged 26 commits into from
Sep 29, 2021

Conversation

antoyo
Copy link
Contributor

@antoyo antoyo commented Jul 18, 2021

This PR introduces a subtree for a gcc-based codegen backend to the repository, per decision in rust-lang/compiler-team#442. We do not yet expect to ship this backend on nightly or run tests in CI, but we do verify that the backend checks (i.e., cargo check) successfully.

Work is expected to progress primarily in https://github.com/rust-lang/rustc_codegen_gcc, with semi-regular upstreaming, like with other subtrees.

@rust-highfive
Copy link
Collaborator

r? @LeSeulArtichaut

(rust-highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 18, 2021
@rust-log-analyzer

This comment has been minimized.

@LeSeulArtichaut
Copy link
Contributor

Really hyped for this! r? @Mark-Simulacrum

@LeSeulArtichaut LeSeulArtichaut added A-gcc Things relevant to the [future] GCC backend T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. A-codegen Area: Code generation labels Jul 18, 2021
@est31
Copy link
Member

est31 commented Jul 19, 2021

The gcc-patches directory is removed in a pending PR so that the patches won't get into this repo.

Not a lawyer, but note that git works in a way that if you clone a given repository, you obtain the entire history. So from my understanding, the GPL patches are still there, and merging this PR might add GPL artifacts to each git clone of the rust repo.

@est31
Copy link
Member

est31 commented Jul 19, 2021

If you really want to remove the directory from git, use git filter-branch or git filter-repo.

@LeSeulArtichaut LeSeulArtichaut added the S-blocked Status: Blocked on something else such as an RFC or other implementation work. label Jul 19, 2021
@bjorn3
Copy link
Member

bjorn3 commented Jul 19, 2021

  • In src/bootstrap/check.rs and src/bootstrap/compile.rs there is a CodegenBackend struct. Currently it only compiles cg_clif, but it is designed to work with arbitrary codegen backends. The check one is probably the easiest to wire up. Just add compiler/rustc_codegen_gcc and rustc_codegen_gcc to the array in should_run and add "gcc" to the array in make_run. For compile you need something similar, but also wire up compilation of libgccjit somehow. Maybe that could be left for a future PR?
  • Next you need to explicitly add or exclude rustc_codegen_gcc from the root cargo workspace. I explicitly excluded rustc_codegen_cranelift to ensure that the Cargo.lock of cg_clif is used instead of the root Cargo.lock to determine the right rev of the cranelift git dependency. I think you will want to do the same thing because of the gccjit.rs git dependency.
  • Finally you will either need to run rustfmt on cg_gcc using rustc's rustfmt.toml and a nightly rustfmt or exclude it in rustfmt.toml.
  • Another thing, once this PR (or any other PR updating the cg_gcc version in this repo) is merged you will want to pull the git subtree to the cg_gcc repo to prevent merge conflicts the next time you pull changes from the rust repo. See my rustup script: https://github.com/bjorn3/rustc_codegen_cranelift/blob/bd2f72f398fa6b609a005cea8ebe16e093937936/scripts/rustup.sh#L29-L53

@joshtriplett
Copy link
Member

If you really want to remove the directory from git, use git filter-branch or git filter-repo.

Or, as another option, you could squash the current git history and just add rustc_codegen_gcc in one commit. (You don't have to do this, but it may be simpler, and I don't think it's critical to preserve the out-of-tree history. New commits could start from the merged version.)

@antoyo antoyo force-pushed the libgccjit-codegen branch from f7c5e19 to c03c298 Compare July 19, 2021 22:33
@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@bors
Copy link
Contributor

bors commented Jul 22, 2021

☔ The latest upstream changes (presumably #87366) made this pull request unmergeable. Please resolve the merge conflicts.

@antoyo antoyo force-pushed the libgccjit-codegen branch from 598516f to a44a134 Compare July 23, 2021 00:01
@rust-log-analyzer

This comment has been minimized.

@antoyo
Copy link
Contributor Author

antoyo commented Jul 23, 2021

So, the CI now pass.

I still need to fix the codegen as the latest changes in rustc made the libcore's test failing.

What else needs to be done for this first PR to land?

Should I run the libcore's test in the CI?
If so, how to a get a patched libgccjit (do I use the binary from the artifacts in my fork of gcc, do I build it in the CI — if so, from my fork or do I need to apply the paches?)?

Apart from that, from the comments above, I believe only the changes to src/bootstrap/compile.rs that @bjorn3 mentioned is missing (should that be included in this PR or a later one?).

@joshtriplett
Copy link
Member

Should I run the libcore's test in the CI?

Per the MCP:

This MCP also proposes gating CI on rustc_codegen_gcc building, but not on it passing any tests.

So, you shouldn't run the testsuite, but you should build the backend.

Long-term, I think we'll want to have a rust-lang/gcc repository that contains a branch with the necessary patches, just as we currently have an LLVM branch at https://github.com/rust-lang/llvm-project/ .

I'd suggest checking with @rust-lang/infra about the best way to handle this. I'll check on Zulip.

@antoyo
Copy link
Contributor Author

antoyo commented Jul 24, 2021

So, you shouldn't run the testsuite, but you should build the backend.

@joshtriplett : By that, do you mean that doing the check is enough, or should I attempt to build it and link to libgccjit?

Have you got any news from asking on Zulip?

Edit: libcore tests are now fixed.

@antoyo
Copy link
Contributor Author

antoyo commented Aug 3, 2021

@LeSeulArtichaut Can you tell me what is blocking this PR? (The GCC patches were removed from the history, btw.)

@glaubitz
Copy link
Contributor

glaubitz commented Aug 3, 2021

@LeSeulArtichaut Can you tell me what is blocking this PR? (The GCC patches were removed from the history, btw.)

Shouldn't the commits above be squashed into a single commit? Seems a bit unusual to keep the history of the development project.

@antoyo
Copy link
Contributor Author

antoyo commented Aug 3, 2021

@LeSeulArtichaut Can you tell me what is blocking this PR? (The GCC patches were removed from the history, btw.)

Shouldn't the commits above be squashed into a single commit? Seems a bit unusual to keep the history of the development project.

I tried doing that, but it stopped being a git subtree. Do you know if there is a way to squash a git subtree without creating an unrelated git history?

@est31
Copy link
Member

est31 commented Aug 3, 2021

@glaubitz same has been done for clippy, miri, rustfmt, rustc-codegen-cranelift, so it's not without precedent. And in the case of the other repos it was way more commits than just 90.

@Mark-Simulacrum
Copy link
Member

Thanks!

What is the process from now on?
Should I check now that the tests still pass to know if I need to update?

I think syncing on a roughly biweekly cadence in both directions is a good idea; the longer you wait the more chance there is for rebase errors to crop up. Of course, if tests are broken or there's some reason to sync more often, there's nothing inherently stopping you from doing so.

@rust-log-analyzer

This comment has been minimized.

@antoyo
Copy link
Contributor Author

antoyo commented Sep 28, 2021

@Mark-Simulacrum Ok, I made the update to today's nightly (some other changes were merged as well), so please review again to get the tests running. Thanks!

@Mark-Simulacrum
Copy link
Member

@bors r+

@bors
Copy link
Contributor

bors commented Sep 28, 2021

📌 Commit 90be409 has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 28, 2021
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Sep 28, 2021
…mulacrum

Libgccjit codegen

This PR introduces a subtree for a gcc-based codegen backend to the repository, per decision in rust-lang/compiler-team#442. We do not yet expect to ship this backend on nightly or run tests in CI, but we do verify that the backend checks (i.e., `cargo check`) successfully.

Work is expected to progress primarily in https://github.com/rust-lang/rustc_codegen_gcc, with semi-regular upstreaming, like with other subtrees.
This was referenced Sep 28, 2021
bors added a commit to rust-lang-ci/rust that referenced this pull request Sep 29, 2021
…laumeGomez

Rollup of 8 pull requests

Successful merges:

 - rust-lang#87260 (Libgccjit codegen)
 - rust-lang#89212 (x.py: run `rustup toolchain link` in setup)
 - rust-lang#89233 (Hide `<...> defined here` note if the source is not available)
 - rust-lang#89235 (make junit output more consistent with default format)
 - rust-lang#89255 (Fix incorrect disambiguation suggestion for associated items)
 - rust-lang#89276 (Fix the population of the `union.impls` field)
 - rust-lang#89283 (Add regression test for issue rust-lang#83564)
 - rust-lang#89318 (rustc_session: Remove lint store from `Session`)

Failed merges:

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit 8642904 into rust-lang:master Sep 29, 2021
@rustbot rustbot added this to the 1.57.0 milestone Sep 29, 2021
@glaubitz
Copy link
Contributor

@antoyo Congratulations! Could you point me to instructions on how to use the new backend? I would like to try building Rust for exotic architectures like Alpha, PA-RISC and Itanium now.

@GuillaumeGomez GuillaumeGomez deleted the libgccjit-codegen branch September 29, 2021 14:06
@bjorn3
Copy link
Member

bjorn3 commented Sep 29, 2021

I believe https://github.com/rust-lang/rustc_codegen_gcc/#how-to-build-a-cross-compiling-libgccjit has the instructions.

@antoyo
Copy link
Contributor Author

antoyo commented Sep 29, 2021

@antoyo Congratulations! Could you point me to instructions on how to use the new backend? I would like to try building Rust for exotic architectures like Alpha, PA-RISC and Itanium now.

There are instructions for building a cross-compiling libgccjit in the readme.
Please note that there are some hacks needed to make this work currently.
But now that this PR has been merged, people can start adding new targets in rustc and the support in the different crates like libc and object.

@glaubitz
Copy link
Contributor

@bjorn3 @antoyo Thanks a lot for all the hints. I will have a look and start working on adding the targets to libc. I already have a draft for m68k here: rust-lang/libc#2414

@antoyo
Copy link
Contributor Author

antoyo commented Sep 29, 2021

@glaubitz I misread what you wrote. If you try to build the rust compiler with this project, it's likely not going to work. It's still work in progress and not all the test suite pass, so it might either produce a non-working rustc or fail to compile it.

dkm added a commit to dkm/misc-builder that referenced this pull request Oct 2, 2021
Make the last revision argument optional for easier local use.

Change git URL for rustc_cg_gcc as it's now part of rust-lang:

 rust-lang/rust#87260

Signed-off-by: Marc Poulhiès <[email protected]>
wip-sync pushed a commit to NetBSD/pkgsrc-wip that referenced this pull request Dec 6, 2021
Pkgsrc changes:
 * Adapt a couple of patches

Upstream changes:

Version 1.57.0 (2021-12-02)
==========================

Language
--------

- [Macro attributes may follow `#[derive]` and will see the original
  (pre-`cfg`) input.][87220]
- [Accept curly-brace macros in expressions, like `m!{ .. }.method()`
  and `m!{ .. }?`.][88690]
- [Allow panicking in constant evaluation.][89508]

Compiler
--------

- [Create more accurate debuginfo for vtables.][89597]
- [Add `armv6k-nintendo-3ds` at Tier 3\*.][88529]
- [Add `armv7-unknown-linux-uclibceabihf` at Tier 3\*.][88952]
- [Add `m68k-unknown-linux-gnu` at Tier 3\*.][88321]
- [Add SOLID targets at Tier 3\*:][86191] `aarch64-kmc-solid_asp3`,
  `armv7a-kmc-solid_asp3-eabi`, `armv7a-kmc-solid_asp3-eabihf`

\* Refer to Rust's [platform support page][platform-support-doc] for more
   information on Rust's tiered platform support.

Libraries
---------

- [Avoid allocations and copying in `Vec::leak`][89337]
- [Add `#[repr(i8)]` to `Ordering`][89507]
- [Optimize `File::read_to_end` and `read_to_string`][89582]
- [Update to Unicode 14.0][89614]
- [Many more functions are marked `#[must_use]`][89692], producing a warning
  when ignoring their return value. This helps catch mistakes such as expecting
  a function to mutate a value in place rather than return a new value.

Stabilised APIs
---------------

- [`[T; N]::as_mut_slice`][`array::as_mut_slice`]
- [`[T; N]::as_slice`][`array::as_slice`]
- [`collections::TryReserveError`]
- [`HashMap::try_reserve`]
- [`HashSet::try_reserve`]
- [`String::try_reserve`]
- [`String::try_reserve_exact`]
- [`Vec::try_reserve`]
- [`Vec::try_reserve_exact`]
- [`VecDeque::try_reserve`]
- [`VecDeque::try_reserve_exact`]
- [`Iterator::map_while`]
- [`iter::MapWhile`]
- [`proc_macro::is_available`]
- [`Command::get_program`]
- [`Command::get_args`]
- [`Command::get_envs`]
- [`Command::get_current_dir`]
- [`CommandArgs`]
- [`CommandEnvs`]

These APIs are now usable in const contexts:

- [`hint::unreachable_unchecked`]

Cargo
-----

- [Stabilize custom profiles][cargo/9943]

Compatibility notes
-------------------

Internal changes
----------------
These changes provide no direct user facing benefits, but represent significant
improvements to the internals and overall performance of rustc
and related tools.

- [Added an experimental backend for codegen with `libgccjit`.][87260]

[86191]: rust-lang/rust#86191
[87220]: rust-lang/rust#87220
[87260]: rust-lang/rust#87260
[88243]: rust-lang/rust#88243
[88321]: rust-lang/rust#88321
[88529]: rust-lang/rust#88529
[88690]: rust-lang/rust#88690
[88952]: rust-lang/rust#88952
[89337]: rust-lang/rust#89337
[89507]: rust-lang/rust#89507
[89508]: rust-lang/rust#89508
[89582]: rust-lang/rust#89582
[89597]: rust-lang/rust#89597
[89614]: rust-lang/rust#89614
[89692]: rust-lang/rust#89692
[cargo/9943]: rust-lang/cargo#9943
[`array::as_mut_slice`]: https://doc.rust-lang.org/std/primitive.array.html#method.as_mut_slice
[`array::as_slice`]: https://doc.rust-lang.org/std/primitive.array.html#method.as_slice
[`collections::TryReserveError`]: https://doc.rust-lang.org/std/collections/struct.TryReserveError.html
[`HashMap::try_reserve`]: https://doc.rust-lang.org/std/collections/hash_map/struct.HashMap.html#method.try_reserve
[`HashSet::try_reserve`]: https://doc.rust-lang.org/std/collections/hash_set/struct.HashSet.html#method.try_reserve
[`String::try_reserve`]: https://doc.rust-lang.org/alloc/string/struct.String.html#method.try_reserve
[`String::try_reserve_exact`]: https://doc.rust-lang.org/alloc/string/struct.String.html#method.try_reserve_exact
[`Vec::try_reserve`]: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.try_reserve
[`Vec::try_reserve_exact`]: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.try_reserve_exact
[`VecDeque::try_reserve`]: https://doc.rust-lang.org/std/collections/struct.VecDeque.html#method.try_reserve
[`VecDeque::try_reserve_exact`]: https://doc.rust-lang.org/std/collections/struct.VecDeque.html#method.try_reserve_exact
[`Iterator::map_while`]: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.map_while
[`iter::MapWhile`]: https://doc.rust-lang.org/std/iter/struct.MapWhile.html
[`proc_macro::is_available`]: https://doc.rust-lang.org/proc_macro/fn.is_available.html
[`Command::get_program`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.get_program
[`Command::get_args`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.get_args
[`Command::get_envs`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.get_envs
[`Command::get_current_dir`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.get_current_dir
[`CommandArgs`]: https://doc.rust-lang.org/std/process/struct.CommandArgs.html
[`CommandEnvs`]: https://doc.rust-lang.org/std/process/struct.CommandEnvs.html
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Jan 22, 2022
Pkgsrc changes:
 * Adjust line numbers in a number of patches
 * remove the --disable-dist-src option, so that we produce
   the rust-src rust component, which we upload to LOCALSRC
   to allow the rust-src package to build, which is needed
   for rust-analyzer.
 * Cargo checksum for vendor/cc no longer needs patching;
   checksum for vendor/libc updated

Upstream changes:

Version 1.57.0 (2021-12-02)
==========================

Language
--------

- [Macro attributes may follow `#[derive]` and will see the original
  (pre-`cfg`) input.][87220]
- [Accept curly-brace macros in expressions, like `m!{ .. }.method()`
  and `m!{ .. }?`.][88690]
- [Allow panicking in constant evaluation.][89508]

Compiler
--------

- [Create more accurate debuginfo for vtables.][89597]
- [Add `armv6k-nintendo-3ds` at Tier 3\*.][88529]
- [Add `armv7-unknown-linux-uclibceabihf` at Tier 3\*.][88952]
- [Add `m68k-unknown-linux-gnu` at Tier 3\*.][88321]
- [Add SOLID targets at Tier 3\*:][86191] `aarch64-kmc-solid_asp3`,
  `armv7a-kmc-solid_asp3-eabi`, `armv7a-kmc-solid_asp3-eabihf`

\* Refer to Rust's [platform support page][platform-support-doc] for more
   information on Rust's tiered platform support.

Libraries
---------

- [Avoid allocations and copying in `Vec::leak`][89337]
- [Add `#[repr(i8)]` to `Ordering`][89507]
- [Optimize `File::read_to_end` and `read_to_string`][89582]
- [Update to Unicode 14.0][89614]
- [Many more functions are marked `#[must_use]`][89692], producing a warning
  when ignoring their return value. This helps catch mistakes such as expecting
  a function to mutate a value in place rather than return a new value.

Stabilised APIs
---------------

- [`[T; N]::as_mut_slice`][`array::as_mut_slice`]
- [`[T; N]::as_slice`][`array::as_slice`]
- [`collections::TryReserveError`]
- [`HashMap::try_reserve`]
- [`HashSet::try_reserve`]
- [`String::try_reserve`]
- [`String::try_reserve_exact`]
- [`Vec::try_reserve`]
- [`Vec::try_reserve_exact`]
- [`VecDeque::try_reserve`]
- [`VecDeque::try_reserve_exact`]
- [`Iterator::map_while`]
- [`iter::MapWhile`]
- [`proc_macro::is_available`]
- [`Command::get_program`]
- [`Command::get_args`]
- [`Command::get_envs`]
- [`Command::get_current_dir`]
- [`CommandArgs`]
- [`CommandEnvs`]

These APIs are now usable in const contexts:

- [`hint::unreachable_unchecked`]

Cargo
-----

- [Stabilize custom profiles][cargo/9943]

Compatibility notes
-------------------

Internal changes
----------------
These changes provide no direct user facing benefits, but represent significant
improvements to the internals and overall performance of rustc
and related tools.

- [Added an experimental backend for codegen with `libgccjit`.][87260]

[86191]: rust-lang/rust#86191
[87220]: rust-lang/rust#87220
[87260]: rust-lang/rust#87260
[88243]: rust-lang/rust#88243
[88321]: rust-lang/rust#88321
[88529]: rust-lang/rust#88529
[88690]: rust-lang/rust#88690
[88952]: rust-lang/rust#88952
[89337]: rust-lang/rust#89337
[89507]: rust-lang/rust#89507
[89508]: rust-lang/rust#89508
[89582]: rust-lang/rust#89582
[89597]: rust-lang/rust#89597
[89614]: rust-lang/rust#89614
[89692]: rust-lang/rust#89692
[cargo/9943]: rust-lang/cargo#9943
[`array::as_mut_slice`]: https://doc.rust-lang.org/std/primitive.array.html#method.as_mut_slice
[`array::as_slice`]: https://doc.rust-lang.org/std/primitive.array.html#method.as_slice
[`collections::TryReserveError`]: https://doc.rust-lang.org/std/collections/struct.TryReserveError.html
[`HashMap::try_reserve`]: https://doc.rust-lang.org/std/collections/hash_map/struct.HashMap.html#method.try_reserve
[`HashSet::try_reserve`]: https://doc.rust-lang.org/std/collections/hash_set/struct.HashSet.html#method.try_reserve
[`String::try_reserve`]: https://doc.rust-lang.org/alloc/string/struct.String.html#method.try_reserve
[`String::try_reserve_exact`]: https://doc.rust-lang.org/alloc/string/struct.String.html#method.try_reserve_exact
[`Vec::try_reserve`]: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.try_reserve
[`Vec::try_reserve_exact`]: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.try_reserve_exact
[`VecDeque::try_reserve`]: https://doc.rust-lang.org/std/collections/struct.VecDeque.html#method.try_reserve
[`VecDeque::try_reserve_exact`]: https://doc.rust-lang.org/std/collections/struct.VecDeque.html#method.try_reserve_exact
[`Iterator::map_while`]: https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.map_while
[`iter::MapWhile`]: https://doc.rust-lang.org/std/iter/struct.MapWhile.html
[`proc_macro::is_available`]: https://doc.rust-lang.org/proc_macro/fn.is_available.html
[`Command::get_program`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.get_program
[`Command::get_args`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.get_args
[`Command::get_envs`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.get_envs
[`Command::get_current_dir`]: https://doc.rust-lang.org/std/process/struct.Command.html#method.get_current_dir
[`CommandArgs`]: https://doc.rust-lang.org/std/process/struct.CommandArgs.html
[`CommandEnvs`]: https://doc.rust-lang.org/std/process/struct.CommandEnvs.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-codegen Area: Code generation A-gcc Things relevant to the [future] GCC backend S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.