Improve TokenCursor.
#107544
Merged
Improve TokenCursor.
#107544
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rustbot
added
S-waiting-on-review
|
Some changes occurred in src/tools/rustfmt cc @rust-lang/rustfmt |
nnethercote
force-pushed
the
improve-TokenCursor
branch
from
02fb0f6 to
a50216a
Compare
|
@bors r+ |
|
📌 Commit a50216ac79c215bbb5ac773cb6c650c8a66397e2 has been approved by It is now in the queue for this repository. |
bors
added
S-waiting-on-bors
|
⌛ Testing commit a50216ac79c215bbb5ac773cb6c650c8a66397e2 with merge 8922ad9d3bc9694b3cb72b9c11bbeeab720023d9... |
|
💔 Test failed - checks-actions |
bors
added
S-waiting-on-review
|
@bors retry CI was broken |
bors
added
S-waiting-on-bors
This comment has been minimized.
This comment has been minimized.
Sometimes the parser needs to desugar a doc comment into `#[doc = r"foo"]`. Currently it does this in a hacky way: by pushing a "fake" new frame (one without a delimiter) onto the `TokenCursor` stack. This commit changes things so that the token stream itself is modified in place. The nice thing about this is that it means `TokenCursorFrame::delim_sp` is now only `None` for the outermost frame.
The motivation here is to eliminate the `Option<(Delimiter, DelimSpan)>`, which is `None` for the outermost token stream and `Some` for all other token streams. We are already treating the innermost frame specially -- this is the `frame` vs `stack` distinction in `TokenCursor`. We can push that further so that `frame` only contains the cursor, and `stack` elements contain the delimiters for their children. When we are in the outermost token stream `stack` is empty, so there are no stored delimiters, which is what we want because the outermost token stream *has* no delimiters. This change also shrinks `TokenCursor`, which shrinks `Parser` and `LazyAttrTokenStreamImpl`, which is nice.
This makes it clear they return token trees, and makes for a nice comparison against `TokenCursor` which returns tokens.
nnethercote
force-pushed
the
improve-TokenCursor
branch
from
a50216a to
a86fc72
Compare
|
@bors r=petrochenkov |
Dylan-DPC
added a commit
to Dylan-DPC/rust
that referenced
this pull request
Feb 3, 2023
…=petrochenkov Improve `TokenCursor`. Some small improvements, for things that were bugging me. Best reviewed one commit at a time. r? `@petrochenkov`
This was referenced Feb 3, 2023
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Feb 3, 2023
Rollup of 6 pull requests Successful merges: - rust-lang#107082 (Autotrait bounds on dyn-safe trait methods) - rust-lang#107427 (Add candidates for DiscriminantKind builtin) - rust-lang#107539 (Emit warnings on unused parens in index expressions) - rust-lang#107544 (Improve `TokenCursor`.) - rust-lang#107585 (Don't cause a cycle when formatting query description that references a FnDef) - rust-lang#107633 (Fix suggestion for coercing Option<&String> to Option<&str>) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some small improvements, for things that were bugging me.
Best reviewed one commit at a time.
r? @petrochenkov