proc-macro derive produced unparseable tokens as of nightly-2019-11-25

[onelson]

I was looking into arabidopsis/typescript-definitions#4 and the pest usage (Parser derive) there works on nightly-2019-11-24, but breaks on nightly-2019-11-25. I'm not familiar with pest, and so I'm unsure if this is an issue specific to the grammar being fed into the proc macro, or something else.

I note that the pest test suite builds and runs fine on both sides of the toolchain boundary, so again, I'm very unsure of where the issue actually lies.

Looking for some direction/advice if you've got it for me. Thanks!

[onelson]

In case it's of interest, I think this is the grammar in question:

WHITESPACE = _{ WHITE_SPACE }

ident = @{ XID_START ~ XID_CONTINUE* }
union = {   item ~ ("|" ~ item)*  }
map = {  "{" ~ "[" ~ "key" ~ ":" ~ key ~ "]" ~ ":" ~ expr ~ "}" }
object = {  "{" ~ (ident ~ ":" ~ expr)? ~ ("," ~ ident ~ ":" ~ expr )* ~ "}" }
tuple = { "[" ~ expr ~ ("," ~ expr )* ~ "]" }
key = { "number" | "string" }
base_type = { "number" | "object" | "string" | "boolean" | "null" }
singleton = { object | map | tuple | base_type | "(" ~ union ~ ")" }
array = { "[]"* }
item = { singleton ~ array  }
expr = { union | "(" ~ expr ~ ")" }
typescript = { SOI ~ expr  ~ EOI }

[CAD97]

Minimal repro:

mod grammar {
    #[derive(::pest_derive::Parser)]
    #[grammar_inline = "
        union = { SOI }
    "]
    struct Parser;
}

cargo +nightly-2019-11-24 build works, cargo +nightly-2019-11-25 fails. Interim workaround: don't use union as a rule name. My hunch is something changed upstream in Rust that makes it register as a keyword in whatever position it's being produced rather than an identifier.

I'm currently attempting to reduce the issue to report upstream. Thanks for reporting this!

(To avoid things like this in the future, we should potentially always emit raw identifiers. This makes sure that any identifier will always work without odd problems, though it might require raw identifier syntax to use. It is allowed to use unnecessary raw identifiers, and is equivalent to the non-raw identifier.)

[onelson]

Glad it was easy to spot!

[CAD97]

I got lucky noticing that you were using a pseudo-keyword and intuiting that was the problem 😛

Even more minimized:

pub enum Rule {
    union,
}

This works on +nightly-2019-11-24 and fails on +nightly-2019-11-25.

Reported upstream as rust-lang/rust#66943

[onelson]

rust-lang/rust#66943 was closed two days ago, and I was able to build using nightly (latest) just now without issue.

Thanks for facilitating the upstream fix!