track incr. comp. dependencies across crates #33476
Conversation
nikomatsakis
force-pushed
the
incr-comp-xcrate
branch
from
0b0b913 to
7e954a0
Compare
| // we walk the predecessors of `MetaData(X)` from the | ||
| // dep-graph. This corresponds to all the inputs that were read to | ||
| // construct the metadata. To create the hash for the metadata, we | ||
| // hash (the hash of) all of those inputs. |
There was a problem hiding this comment.
This is nice indeed, I like it.
|
☔ The latest upstream changes (presumably #33091) made this pull request unmergeable. Please resolve the merge conflicts. |
Since that has landed already, could you remove those commits from this PR? That would also make it easier to review since Github won't show a merged diff because there are too many changed files. |
|
@michaelwoerister will do so once this flight lands :) |
nikomatsakis
force-pushed
the
incr-comp-xcrate
branch
2 times, most recently
from
7aa973e to
1853f26
Compare
|
@michaelwoerister rebased :) |
|
☔ The latest upstream changes (presumably #33443) made this pull request unmergeable. Please resolve the merge conflicts. |
nikomatsakis
force-pushed
the
incr-comp-xcrate
branch
from
1853f26 to
a8bfbc4
Compare
| // node each time we pull the information for an item out of the | ||
| // crate store. | ||
| MetaData(D), | ||
|
|
||
| // Represents different phases in the compiler. | ||
| CrateReader, | ||
| CollectLanguageItems, |
There was a problem hiding this comment.
This commit looks good.
|
☔ The latest upstream changes (presumably #33425) made this pull request unmergeable. Please resolve the merge conflicts. |
|
OK, so I was able to capture a stack trace in VMWare (this was...not easy). It had nothing to do w/ a stack overflow. Rather, I think that the error code I was seeing was being produced by the |
This got removed at some point, it seems.
Conflicts: src/librustc_metadata/csearch.rs
This lets us determine what was used to construct the metadata. Conflicts: src/librustc_metadata/encoder.rs
This way, multiple compilations can share the same work directory.
Generate a second hash file that contains the metadata for an X node.
we need them to be able to retrace.
This is needed for incremental compilation harness to support cross-crate testing. Also support cfg for typechecking prettyprint
We used to store a u64 converted to a String for some reason. Now we don't.
Create some re-usable filtering subroutines.
For external crates, we must build up a map that goes from the DefKey to the DefIndex. We do this by iterating over each index that is found in the metadata and loading the associated DefKey.
This commit reorganizes how the persist code treats hashing. The idea is that each crate saves a file containing hashes representing the metadata for each item X. When we see a read from `MetaData(X)`, we can load this hash up (if we don't find a file for that crate, we just use the SVH for the entire crate). To compute the hash for `MetaData(Y)`, where Y is some local item, we examine all the predecessors of the `MetaData(Y)` node and hash their hashes together.
In MSVC (at least), linking requires accessing metadata, which generates reads.
nikomatsakis
force-pushed
the
incr-comp-xcrate
branch
from
0351a6c to
f860f8b
Compare
|
@bors r=mw |
|
📌 Commit f860f8b has been approved by |
|
Wow, that sounds super annoying! 😬 |
track incr. comp. dependencies across crates This PR refactors the compiler's incremental compilation hashing so that it can track dependencies across crates. The main bits are: - computing a hash representing the metadata for an item we are emitting - we do this by making `MetaData(X)` be the current task while computing metadata for an item - this naturally registers reads from any tables and things that we read for that purpose - we can then hash all the inputs to those tables - tracking when we access metadata - we do this by registering a read of `MetaData(X)` for each foreign item `X` whose metadata we read - hashing metadata from foreign items - we do this by loading up metadata from a file in the incr. comp. directory - if there is no file, we use the SVH for the entire crate There is one very simple test only at this point. The next PR will be focused on expanding out the tests. Note that this is based on top of #33228 r? @michaelwoerister
This PR refactors the compiler's incremental compilation hashing so that it can track dependencies across crates. The main bits are:
MetaData(X)be the current task while computing metadata for an itemMetaData(X)for each foreign itemXwhose metadata we readThere is one very simple test only at this point. The next PR will be focused on expanding out the tests.
Note that this is based on top of #33228
r? @michaelwoerister