Do we have a central document with the Rust grammar?

[icorderi]

Hey guys, we are trying to revamp the CodeMirror definition for Rust given that it is a bit out of date and doesn't parse Rust code correctly anymore. Here is the issue on CodeMirror.

Do we have a document outlining the Rust language grammar? I've seen bits and pieces all over the reference document.

Who in the Rust community can help with this?

[steveklabnik]

There isn't. One would be lovely.

This is a superset of #9022, really.

[huonw]

There's a lexical grammar in https://github.com/rust-lang/rust/tree/master/src/grammar . It's not currently guaranteed to be up-to-date.

[icorderi]

@steveklabnik should we start an effort to get a document going with all the grammar?

[steveklabnik]

Please.

[icorderi]

@steveklabnik, @huonw Ok, how do we go about this?
Going through the reference and building a grammar.md with the same sections but keeping only the AST content sounds like a good start? We can leave a link back on each big chunk of the AST pointing to the corresponding reference section.

For example:

4.1 Macros (more details)

expr_macro_rules : "macro_rules" '!' ident '(' macro_rule * ')' ;
macro_rule : '(' matcher * ')' "=>" '(' transcriber * ')' ';' ;
matcher : '(' matcher * ')' | '[' matcher * ']'
        | '{' matcher * '}' | '$' ident ':' ident
        | '$' '(' matcher * ')' sep_token? [ '*' | '+' ]
        | non_special_token ;
transcriber : '(' transcriber * ')' | '[' transcriber * ']'
            | '{' transcriber * '}' | '$' ident
            | '$' '(' transcriber * ')' sep_token? [ '*' | '+' ]
            | non_special_token ;

What do you think?

[steveklabnik]

Ideally, go through the code and figure it out. 😅 It's possible reference.md has diverged.

The full grammar should be part of the reference.

[icorderi]

Yea, reference.md has a mix of grammar, examples and a lot of interesting text to read. I'll go ahead and create a grammar.md then, and place it under the same directory.

We can figure out later how to have the reference.md reference --for lack of a better word-- the corresponding snippets of grammar directly from grammar.md.

Sounds good?

[icorderi]

@huonw @steveklabnik I have this grammar.md going on my fork. I left the intro as it was given that is talks about the grammar... perhaps it's not necessary to keep in that document. If someone is looking at a grammar for a language you could assume they understand how to read the productions.

The relevant pieces start here. I cleaned up to "CHECKPOINT". There are a lot of features with no grammar definition (to mention a few: type, trait, struct, enum, fn... are all missing).

If you guys think this is worth having, I'll continue cleaning it up, leaving only the existing grammar bits from the reference.md. But that is just the beginning, then comes the task to fill in all the blanks.

[steveklabnik]

Nice! This is looking awesome. I just skimmed it, but it's certainly in the right direction.

[icorderi]

Pinging @marijnh from CodeMirror to keep him up to date on this effort.

[icorderi]

Ok, here is the new grammar.md file with everything that was available at the reference.md.
Should I do a PR with the current state?
It might be cleaner to manage individual issues for each of the big missing chunks if we pull in more people for this...

• Functions
• Structures
• Traits
• Implementations
• Operators
• Statements
• Expressions

Who from rust side can help fill in the blanks? I can certainly do a first pass based on examples, but I'm not familiar with everything that might be possible.

[Gankra]

If you're uncertain. You can still start a WIP PR for review and iteration.

[brson]

The makefiles may still need to be updated to render the docs, and a link placed somewhere.

[marijnh]

Assuming that http://doc.rust-lang.org/grammar.html reflects the current state of the grammar, there's still a lot of FIXMEs and missing rules. Is there an issue tracking the completion of an actual full grammar?

[pnkfelix]

@marijnh I agree that having an issue for the full grammar is important, so I filed #22445.

(I chose spawning a new issue over reopening this ticket, because I think the most interesting part of this ticket was choosing an tool for documenting the grammar and incorporating an invocation of that tool into our build process, yay!)