Stabilize reserved prefixes

[nikomatsakis]

Reserved prefixes stabilization report

Links

• RFC
• Tracking issue
• Rust reference PR -- missing?
• Implementation PR

Summary

• any_identifier#, any_identifier"...", and any_identifier'...' are now reserved
  syntax, and no longer tokenize.
• This is mostly relevant to macros. E.g. quote!{ #a#b } is no longer accepted.
• It doesn't treat keywords specially, so e.g. match"..." {} is no longer accepted.
• Insert whitespace between the identifier and the subsequent #, ", or '
  to avoid errors.
• Edition migrations will help you insert whitespace in such cases.

Details

To make space for new syntax in the future, we've decided to reserve syntax for prefixed identifiers and literals: prefix#identifier, prefix"string", prefix'c', and prefix#123, where prefix can be any identifier. (Except those prefixes that already have a meaning, such as b'...' (byte strings) and r"..." (raw strings).)

This provides syntax we can expand into in the future without requiring an edition boundary. We may use this for temporary syntax until the next edition, or for permanent syntax if appropriate.

Without an edition, this would be a breaking change, since macros can currently accept syntax such as hello"world", which they will see as two separate tokens: hello and "world". The (automatic) fix is simple though: just insert a space: hello "world". Likewise, prefix#ident should become prefix #ident. Edition migrations will help with this fix.

Other than turning these into a tokenization error, the RFC does not attach a meaning to any prefix yet. Assigning meaning to specific prefixes is left to future proposals, which will now—thanks to reserving these prefixes—not be breaking changes.

Some new prefixes you might potentially see in the future (though we haven't
committed to any of them yet):

• k#keyword to allow writing keywords that don't exist yet in the current edition. For example, while async is not a keyword in edition 2015, this prefix would've allowed us to accept k#async in edition 2015 without having to wait for edition 2018 to reserve async as a keyword.
• f"" as a short-hand for a format string. For example, f"hello {name}" as a short-hand for the equivalent format!() invocation.
• s"" for String literals.
• c"" or z"" for null-terminated C strings.

How unresolved questions were resolved and other interesting developments

Where and how to enforce prefixes

The biggest question was where to enforce the prefixes and emit errors. We ultimately opted to emit errors in the lexer, which meant that the lexer had to become aware of the current edition. There was an alternative of using "jointness" and enforcing the conditions in the parser. The idea was to leverage the fact that Rust tokens (at least some subset of them) record not only their content but whether they are separated by whitespace from the next token. This was intended to enable compound operators like << to be parsed as two < tokens in some parts øf the parser (types) and as a single token elsewhere (expressions), without the lexer having to know what state the parser was in. This same approach could conceptually be used so that the lexer doesn't have to know the edition.

As described in detail in this writeup, however, the jointness approach had several downsides. For example, it meant that lexing of literals was independent of prefix: we might like f"{foo("bar")}" to be lexed a a string, but that is not possible unless the lexer knows that an f string can contain embedded expressions. Similarly, which escape codes the lexer accepts depends on the prefix (e.g. \x for b""). (This is especially relevant for raw strings: whether fr"\" is accepted or not depends on what meaning we assign to fr.) Jointness also had forwards compatbility hazards with macro arm ordering. Finally, the lexer-based approach can be converted to a jointness-based approach later, as it currently gives errors much earlier in the process.

There were also advantages to jointness: it would allow more procedural macro prototyping, and it means that the lexer would remain independent of edition.

Edition used for procedural macro APIs

There are some procedural macro APIs that lex tokens from strings. Those APIs have not traditionally taken a span or other information from which an edition can be derived. Those APIs will be documented with the Edition that they use to do lexing. In the future we may wish to add new APIs that take a Span or other parameter and use that to derive the Edition.

[nikomatsakis]

@rfcbot fcp merge

I propose that we stabilize reserved prefixes. Note that they will not be exposed to stable completely until Rust 2021 is stabilized.

[rfcbot]

Team member @nikomatsakis has proposed to merge this. The next step is review by the rest of the tagged team members:

• @cramertj
• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm

Concerns:

• no-reference-PR resolved by Stabilize reserved prefixes #88140 (comment)

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[nikomatsakis]

@rfcbot concern no-reference-PR

We need to describe these changes in the Rust reference!

[m-ou-se]

Edition used for procedural macro APIs

There are some procedural macro APIs that lex tokens from strings. Those APIs have not traditionally taken a span or other information from which an edition can be derived. Those APIs will be documented with the Edition that they use to do lexing. In the future we may wish to add new APIs that take a Span or other parameter and use that to derive the Edition.

proc_macro::TokenStream::from_str and proc_macro::Literal::from_str are affected, and now use the edition of the proc macro definition. (They use the call_span for all tokens they produce, which carries the source location of the invocation, but the edition of the proc macro definition.)

This affects quote::quote!{}, which will have to be updated to Rust 2021 before you can use prefixed things with it. That's not relevant right now since all the prefixes are reserved anyway. (It becomes relevant once we add z"" or something.)

[m-ou-se]

This FCP needs to start today to make it in time for 1.56.

[m-ou-se]

We need to describe these changes in the Rust reference!

@nikomatsakis I don't think that needs to block the start of the FCP, right? Can you resolve your concern? We can handle the reference while the FCP is in progress.

[nikomatsakis]

@rfcbot resolve no-reference-PR

I'm going to mark this as resolved for now but try to get this done ASAP

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

[joshtriplett]

AFAICT this is done.