in declarative macros, "literal" matching arm can throw compiler error rather than matching or failing to match

[edwardpeters]

(I'm not confident enough to call this a bug but I think it qualifies as unexpected behavior)

I tried this code:

macro_rules! show_me{
    ($l : literal $($tail : tt)*) => {
        println!("{} is a literal followed by {}", stringify!($l), stringify!($($tail)*));
        show_me!($($tail)*)
    };
    ($token : tt $($tail : tt)*) => {
        println!("{} is something else followed by {}", stringify!($token), stringify!($($tail)*));
        show_me!($($tail)*)
    };
    () => {}
}

fn main() {
    show_me!(-x);
}

I expected that this code would see no literals in the passed expression, and then fall through to the next branch, outputing:

- is something else followed by x
x is something else followed by 

(note that it does this for show_me!(+x))

Instead, the compiler fails with:

error: unexpected token: `x`
  --> src/main.rs:20:15
   |
8  |     ($l : literal $($tail : tt)*) => {
   |      ------------ while parsing argument for this `literal` macro fragment
...
20 |     show_me!(-x);
   |               ^

rustc --version --verbose

rustc 1.66.0 (69f9c33d7 2022-12-12)
binary: rustc
commit-hash: 69f9c33d71c871fc16ac445211281c6e7a340943
commit-date: 2022-12-12
host: x86_64-unknown-linux-gnu
release: 1.66.0
LLVM version: 15.0.2

The behavior is the same in nightly build:
rustc --version --verbose

rustc 1.68.0-nightly (659e169d3 2023-01-04)
binary: rustc
commit-hash: 659e169d37990b9c730a59a96081f2ef7afbe8f1
commit-date: 2023-01-04
host: x86_64-unknown-linux-gnu
release: 1.68.0-nightly
LLVM version: 15.0.6

And beta:
rustc --version --verbose

rustc 1.67.0-beta.6 (51b03459a 2022-12-31)
binary: rustc
commit-hash: 51b03459a49d03dbad7d120fb8575fc4580c057b
commit-date: 2022-12-31
host: x86_64-unknown-linux-gnu
release: 1.67.0-beta.6
LLVM version: 15.0.6

[Noratrieb]

The reference states that a literal metavariable may match something starting with a -: https://doc.rust-lang.org/reference/macros-by-example.html#metavariables

The section you are quoting is about the behavior of the tt metavariable matcher, where a single $x:tt will only match the - and not the literal behind it.

[edwardpeters]

Okay, I'll withdraw the bit about the documentation - but I still think it's unexpected and undesirable that the matcher blow up compilation rather than fail to match. (It seems to do that for some other cases as well, but I haven't been able to find what the rule is for what will not match vs. what will blow up on an attempted match.)

[Noratrieb]

The rule for macro matching is that when encountering a metavariable (also called nonterminal), it looks whether the first token of the input is allowed as a start to that nonterminal. A - is allowed as the start of a literal. If the token is allowed, it commits to parsing the nonterminal and the whole arm.

This behavior is important for forwards compatibility. This allows us to extend syntax (especially something like expression syntax) without breaking existing macros. If this wasn't the case, you could write macros that go to the next arm because it was an invalid expression, but in the future if we made it a valid expression, behavior would change.

Maybe you're interested in #103534 which discusses some of the matching semantics

[edwardpeters]

@Nilstrieb That makes sense, thanks for the explanation.