Skip to content

Commit

Permalink
Fix macro matching for NoDelim delimited sequences.
Browse files Browse the repository at this point in the history
PR rust-lang#95159 unintentionally changed the behaviour of declarative macro
matching for `NoDelim` delimited sequences.
- Before rust-lang#95159, delimiters were implicit in `mbe::Delimited`. When
  doing macro matching, normal delimiters were generated out of thin air
  as necessary, but `NoDelim` delimiters were not. This was done within
  `TokenTree::{get_tt,len}`.
- After rust-lang#95159, the delimiters were explicit. There was an unintentional
  change whereby `NoDelim` delimiters were represented explicitly just
  like normal delimeters.
- rust-lang#95555 introduced a new matcher representation (`MatcherLoc`) and the
  `NoDelim` delimiters were made explicit within it, just like
  `mbe::Delimited`.
- rust-lang#95797 then changed `mbe::Delimited` back to having implicit
  delimiters, but because matching is now being done on `MatcherLoc`,
  the behavioural change persisted.

The fix is simple: remove the explicit `NoDelim` delimiters in the
`MatcherLoc` representation. This gives back the original behaviour.

As for why this took so long to be found: it seems that `NoDelim`
sequences are unusual. It took a macro that generates another macro
(from the `yarte_lexer` crate, found via a crater run) to uncover this.

Fixes rust-lang#96305.
  • Loading branch information
nnethercote committed Apr 22, 2022
1 parent b04c532 commit fe549f1
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions compiler/rustc_expand/src/mbe/macro_parser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,13 @@ pub(super) fn compute_locs(matcher: &[TokenTree]) -> Vec<MatcherLoc> {
let close_token = Token::new(token::CloseDelim(delimited.delim), span.close);

locs.push(MatcherLoc::Delimited);
locs.push(MatcherLoc::Token { token: open_token });
if delimited.delim != token::DelimToken::NoDelim {
locs.push(MatcherLoc::Token { token: open_token });
}
inner(&delimited.tts, locs, next_metavar, seq_depth);
locs.push(MatcherLoc::Token { token: close_token });
if delimited.delim != token::DelimToken::NoDelim {
locs.push(MatcherLoc::Token { token: close_token });
}
}
TokenTree::Sequence(_, seq) => {
// We can't determine `idx_first_after` and construct the final
Expand Down

0 comments on commit fe549f1

Please sign in to comment.