Tracking issue for syn 1.0

[dtolnay]

Current prerelease: syn-next = "1.0.0-rc4"
Release notes in progress:

---

Syn is a library for parsing Rust code, largely geared toward use in procedural macros but generally applicable as a Rust parser outside of that too.

Syn was originally a sidequest on the effort to making Serde's derive macros work on stable Rust in 2016, but has since taken off as a foundational library in its own right. These days it appears in support of diverse use cases like:

• convenient interop between Rust-based Wasm modules and JavaScript,
• easy command line interfaces,
• a polished async experience, and
• a concurrency framework for embedded real time systems.

Excitingly, the fraction of open source Rust projects that rely on Syn is still climbing aggressively (see red line). This reflects how important procedural macros are to the experience we want people to have when working with Rust — empowering everyone to build reliable and efficient software.

This 1.0 release signifies that Syn is a polished and stable library and that it offers a user experience we can stand behind as the way we recommend for all Rustaceans to write procedural macros.

Be aware that the underlying Rust language will continue to evolve. Syn is able to accommodate most kinds of Rust grammar changes via the nonexhaustive enums and Verbatim variants in the syntax tree, but we will plan to put out new major versions on a 12 to 24 month cadence to incorporate ongoing language changes as needed.

Note: in combination with this release, please see also the release notes for quote 1.0 which resolves multiple longstanding limitations of the quote macro.

Breaking changes

• Minimum required Rust version is raised from rustc 1.15 to 1.31.

Items

• The syntax tree for function signatures has been redesigned. The types MethodSig and FnDecl have been unified into one type Signature that is common across all varieties of function signatures.

• The syntax tree for function arguments has been redesigned. The new type is FnArg which represents one argument of a function, associated function, trait function, or foreign function. Arguments of closures are no longer treated as FnArg and are simply a Pat instead.

• The Fields type now implements IntoIterator, in addition to the previously existing IntoIterator impls on &Fields and &mut Fields. This may require changing .into_iter() calls to .iter() when used on a value of type &Fields.

• The representation of Item::Macro2 2.0-style declarative macros have been collapsed into a single token stream pending figuring out what the syntax should be in rust-lang/rust#39412.

• Item::Existential and ImplItem::Existential have been removed in favor of RFC 2515.

Expressions

• Expr::Await is a new variant of expression for postfix .await syntax per ongoing work on rust-lang/rust#50547.

• Expr::InPlace has been removed as a result of RFC 2387.

Types

• BareFnArgName has been eliminated in favor of Ident for representing the arguments of a Type::BareFn.

Patterns

• Pat::Or is a new variant of pattern as specified by RFC 2535.

• Pat::Type is a new variant of pattern for type ascription as specified by RFC 2522.

• Pat::Rest is a new variant of pattern representing the .. pattern as specified by RFC 2707.

• The Pat::Slice and Pat::Tuple patterns have been redesigned to accomodate RFC 2359.

• Pat::Ref has been renamed to Pat::Reference to align with Expr::Reference and Type::Reference.

Tokens

• The Lit::Int and Lit::Float literal types have been redesigned to represent arbitrarily large numbers and arbitrary suffixes. Rather than .value(), there is a .base10_digits() accessor which the caller should parse into the appropriate representation.

• The types of the token representation of the self and Self tokens have been renamed from Self_ / CapSelf to SelfValue / SelfType respectively. This does not affect you if you are referring to them as Token![self] and Token![Self].

• The Lit::Verbatim literal escape hatch now holds a proc_macro2::Literal primitive token directly, rather than the previous LitVerbatim wrapper.

Attributes

• The Attribute::interpret_meta method is removed in favor of Attribute::parse_meta which produces a better error.

• The variants of Meta now contain a Path rather than a single Ident as specified by RFC 2103.

• Function parameters now hold attributes as specified by RFC 2565.

More

• The argument of Path::is_ident is now taken by reference. The common case of using this method with a string literal will continue to work: path.is_ident("...").

• The type returned by Punctuated::into_iter no longer has a P type parameter, as it only returns sequence elements without punctuation.

• The first, last, and last_mut accessors of Punctuated now return the element T only, no longer a punctuated pair. The old paired behavior of .first() can be written as .pairs().next(), .last() can be written as .pairs().next_back(), and .last_mut() can be written as .pairs_mut().next_back().

• Various uses of tts as a field name in the syntax tree have been renamed to tokens to eliminate the nonstandard abbreviation.

[dtolnay]

FYI @mystor @alexcrichton @SergioBenitez @sgrif.

The Syn changes are minor so it should be fine not to bother with the rc versions this time, but I figured I would give some advance notice. The quote changes are potentially interesting. Real release is scheduled for the week of August 12.

[Centril]

• The variants of Meta now contain a Path rather than a single Ident as specified by RFC 2103.

The definition of Meta doesn't seem to support rust-lang/rust#57367 (comment)? It would be good to have a translation from Attribute to that grammar in some typed fashion.

Be aware that the underlying Rust language will continue to evolve. Syn is able to accommodate most kinds of Rust grammar changes via the Verbatim variants in the syntax tree, but we will plan to put out new major versions on an 8 to 12 month cadence to incorporate ongoing language changes in a first class way.

I would strongly recommend waiting for #[non_exhaustive] in some form to enable a more well typed gradual incorporation of language changes.

[dtolnay]

The definition of Meta doesn't seem to support rust-lang/rust#57367 (comment)? It would be good to have a translation from Attribute to that grammar in some typed fashion.

Thanks for the link! In my opinion that grammar doesn't have enough structure to benefit from a more structured representation than Attribute. Meta are a subset of attributes that are easier to work with in exchange for restricting yourself to a subset of the grammar that is common by convention. Use cases beyond that subset can parse what they need from Attribute.

I would strongly recommend waiting for #[non_exhaustive] in some form to enable a more well typed gradual incorporation of language changes.

Noted; I've been considering it but would use a __Nonexhaustive variant instead of waiting because it is effectively the same thing but stable.

[Centril]

Thanks for the link! In my opinion that grammar doesn't have enough structure to benefit from a more structured representation than Attribute. Meta are a subset of attributes that are easier to work with in exchange for restricting yourself to a subset of the grammar that is common by convention. Use cases beyond that subset can parse what they need from Attribute.

It's sort of the point of rust-lang/rust#57367 (comment) to change the convention (which mainly exists due to language limitations), e.g.:

• #[serde(bound = "T: MyTrait")]
• #[serde(default = "Priority::lowest")]
• #[proptest(value = "None")]

...into something better (which notably does not embed strings into the EDSL):

• #[serde(bound(T: MyTrait))]
• #[serde(default(Priority::lowest))]
• #[proptest(value(None))]

The way I'd do it is to be more lazy about MetaList, expose:

pub struct MetaParens {
    pub ident: Ident,
    pub paren_token: Paren,
    pub tokens: TokenStream,
}

and then add a way to convert MetaParens to MetaList.

[dtolnay]

It ends up not being a net benefit. Once you've accounted for formats like #[serde(default = Priority::Low, rename = "pri")], there ends up being so little generalizable structure to attributes that manipulating them as Meta is no longer any easier than writing the parser without Meta. This isn't a reflection of your MetaParens suggestion being bad; instead it reflects how well suited the Parse trait already is to handling such things. So while we could lazify and complicate Meta to fit 95% of use cases with a user experience that is 0-10% easier than the Parse trait, the current tradeoff fits 85% of use cases with a user experience that is substantially simpler.

[alexcrichton]

Thanks for championing this @dtolnay, this all sounds great to me!

There's two things I was wondering about as well:

we will plan to put out new major versions on a 9 to 18 month cadence to incorporate ongoing language changes as needed

I understand the need for this but if this were applied to serde, for example, it'd be pretty disruptive. As a massively used dependency it seems like we should try to bend over backwards pretty hard to see if we can avoid doing breaking changes for as long as possible.

Would it be possible to make all the enums of syn non-exhaustive and still preserve the basic ergonomics of what we have today? For example folds would still work, matches would have to return errors but that seems ok, and otherwise I'm not sure how too too disruptive this would be otherwise.

I suspect though that this has been thought about, so I'm curious to hear sort of what y'all have been thinking about this.

Minimum required Rust version is raised from rustc 1.15 to 1.31.

Why not 1.32.0? (being able to use the 2018 edition seems like it'd be nice?)

[dtolnay]

I understand the need for this but if this were applied to serde, for example, it'd be pretty disruptive.

This is true but it's only because serde appears in the public API of so many libraries. By contrast there are almost no libraries with syn in the public API, basically synstructure and a small handful of others. A lot of crates depend on syn but those are all proc macros with only macros in the API where a syn upgrade can be done transparently.

Analyzing changes from past releases, having a policy of non-exhaustive everything would not noticeably reduce the rate at which language changes beyond the add-a-variant variety stack up.

Why not 1.32.0? (being able to use the 2018 edition seems like it'd be nice?)

We will be using 2018 edition with 1.31.

[alexcrichton]

Ah ok, that's a good point that it's not just new variants but tweaks to existing ones. If that's the case I think that there's no choice but to plan for eventual new breaking releases.

This point is moot, but although I agree that it's less pressing because syn isn't in the public API it's arguably still 'just as disruptive' in the sense that there will be a lot of effort necessary to upgrade parsers and avoid compiling 5 versions of syn in one crate graph eventually. This work can all largely happen independently because there's no public API dependencies though.

We will be using 2018 edition with 1.31.

Oops, my mistake!

[dtolnay]

I released 1.0.0-rc4 and don't have any further changes planned, so if nothing else comes up then this will be the release going out this week, likely Tuesday.

[dtolnay]

https://github.com/dtolnay/syn/releases/tag/1.0.0