Refactoring attributes in the compiler

[jdonszelmann]

While working on #125418 with @m-ou-se, I've interacted quite a bit with attributes in the compiler. I've got some thoughts about the way they currently work. I'm posting this as a mix between an explanation of the status quo and why I think that's an issue, in addition to also serving as a kind of tracking issue for these changes if I've convinced you that this is a problem.

Quick Overview

From the ground up: There are several syntaxes for macros, one of those syntaxes is attributes which can have several forms. Attributes can be expanded, either as a user defined attribute macro, or as an "active" built in attribute like #[test]. However, some attributes are kept around for the entire compilation lifecycle.

These built-in attributes are never expanded. Instead, they are kept around and serve as markers or metadata to guide the compilation process at various stages. There are currently around 100 of these.

The problem

While most of what is parsed, is later lowered during rustc_ast_lowering, attributes are not, mostly. Many crates under compiler/ depend on rustc_ast just to use ast::Attribute. Let's see what that means:

Partial lowering and impossible states

One part of attributes actually is lowered, attributes of the form #[key = "value"] aka MetaNameValueStr. To be able to do that, the ast contains an enum AttrArgsEq that already has a variant for when eventually it is lowered:

rust/compiler/rustc_ast/src/ast.rs

Lines 1697 to 1700 in 11ee3a8

	pub enum AttrArgsEq {
	Ast(P<Expr>),
	Hir(MetaItemLit),
	}

For one part of the compilation process, the Ast variant is always active and Hir is completely unused, while later in the compiler the reverse is true. In some places people didn't realize this and they provided implementations for both cases while only one could occur,
while in other places they are marked as unreachable, like here:

rust/compiler/rustc_ast/src/visit.rs

Line 1241 in 11ee3a8

unreachable!("in literal form when walking mac args eq: {:?}", lit)

Another case of partial lowering is the tokens field:

rust/compiler/rustc_ast_lowering/src/lib.rs

Line 948 in 11ee3a8

tokens: None,

Which is later extensively defended against, making sure this really happened:

rust/compiler/rustc_query_system/src/ich/impls_syntax.rs

Lines 41 to 54 in 11ee3a8

	debug_assert!(!attr.ident().is_some_and(|ident| self.is_ignored_attr(ident.name)));
	debug_assert!(!attr.is_doc_comment());
	let ast::Attribute { kind, id: _, style, span } = attr;
	if let ast::AttrKind::Normal(normal) = kind {
	normal.item.hash_stable(self, hasher);
	style.hash_stable(self, hasher);
	span.hash_stable(self, hasher);
	assert_matches!(
	normal.tokens.as_ref(),
	None,
	"Tokens should have been removed during lowering!"
	);
	} else {

Parse, don't validate.

I'm a big fan of the blog post Parse, don't validate. Generally rust's type system makes this pattern the most obvious thing to do and it's what I teach my university students every year. However, that is exactly what we aren't doing with attributes. In rustc_passes/check_attr.rs we first validate extensively, and emit various diagnostics. However, every single attribute is later parsed again where it is needed. I started making a small overview, but 100 attributes is a lot

But basically, of the first 19 attributes I looked at, 5 are Word attributes and trivial, a few are parsed together, but in total I've found 11 completely distinct and custom parsing logics, not reusing any parts, spread over as many files and compiler crates.

I lied a little there, the parsing does reuse some things. For example, the attributes are turned into MetaItems using common logic. However, that doesn't change the fact that attributes are effectively re-validated scattered around the compiler, and many of these places have more diagnostics of their own, that could've happened during the earlier validation. It also means that at a very late stage in the compiler, we are still dealing with parsing TokenStreams, something that you'd think we should abstract away a little after parsing.

An example of such custom parsing logic:

rust/compiler/rustc_middle/src/ty/context.rs

Lines 1447 to 1469 in 11ee3a8

	let get = |name| {
	let Some(attr) = self.get_attr(def_id, name) else {
	return Bound::Unbounded;
	};
	debug!("layout_scalar_valid_range: attr={:?}", attr);
	if let Some(
	&[
	ast::NestedMetaItem::Lit(ast::MetaItemLit {
	kind: ast::LitKind::Int(a, _),
	..
	}),
	],
	) = attr.meta_item_list().as_deref()
	{
	Bound::Included(a.get())
	} else {
	self.dcx().span_delayed_bug(
	attr.span,
	"invalid rustc_layout_scalar_valid_range attribute",
	);
	Bound::Unbounded
	}
	};

Flexibility

Finally, though I have fewer concrete examples of this, sticking to ast::Attribute throughout the compiler removes quite some flexibility. Everything has to fit into an ast::Attribute, or if it doesn't, you'd have to create more variants like AttrArgsEq::Hir to support something in the ast that shouldn't even be part of the ast, forcing you to add a myriad of exceptions in parts of the compiler where such an extra variant isn't relevant yet. Specifically, for #125418 we noticed this because we wanted to do some limited form of name resolution for a path stored in an attribute, which proved next to impossible.

Ideas

1. Lower attributes during rustc_ast_lowering.

I've got 90% of a commit ready to do this, and it's what sparked the idea for this issue. It leads to some code duplication. I'm a little unhappy about it, because it forces a lot of changes across the entire compiler, exactly because attribute parsing now happens in so many places. However, it already means that a lot of assertions can be removed because at some part of the compiler, the fact that an Attribute can't have certain fields and values anymore becomes encoded in the type system. I'll open a PR for this soon, and we can discuss whether we think this is a good first step.

What also doesn't help is that rustc_attr currently has logic to validate attributes, but these functions are called in wildly different parts of the compiler. Some functions here validate actual ast::Attributes from before lowering, while other functions validate new hir::Attributes. Bugs here seem easy to make, since even though currently these are the same type, they don't always contain the same fields....

2. The "real solution": parse, don't validate

As I see it, what would make attributes so much nicer to work with, is if there was a place in the compiler (something like the rustc_attr crate, but actually good) where all attributes are turned from their ast tokeny representation into some specific attribute representation. Something like the following, based on the examples I've looked at in the table I showed a little higher up:

enum InlineKind {
    Always,
    Never,
    Normal
}

enum Attribute {
    Diagnostic {
        message: Symbol,
        name: Symbol,
        notes: Vec<Symbol>
    },
    Inline(InlineKind),
    Coverage(bool),
    // ...
}

This structure contains only the information necessary to use each attribute, and all the diagnostics happen while parsing into this structure. That has the added benefit that this datastructure itself serves as great documentation as to what values an attribute allows. It's super clear here that a #[diagnostic] attributes contains a message, name and some notes. Currently, you'd have to make sure the written documentation for this attribute is up-to-date enough.

The translation from ast::Attribute to this new parsed attribute should, I think, happen during AST to HIR lowering.

I think the advantages of this should be pretty obvious, based on the examples I've given of the problems with the current approach. However, I could think of some potential blockers people might care about:

• some errors might now be thrown in different stages of the compiler (earlier). I'd say this can actually be an advantage, but I've not talked to enough people to know whether this is a problem anywhere
• A file with code to parse these attributes will contain code of many different features. I personally like that, but it also means that those features themselves become slightly less self-contained.
• Validity of attributes still needs to be checked on-site. (like track_caller not being valid on closures, given certain feature flags)
• Affects large parts of the compiler, also unstable parts, where people are actively opening merge requests and might run into conflicts.

Part two I have not worked on personally. I might, if I find enough time, but if someone feels very inspired to pick this up or lead this (or tell my why this is a dumb idea) feel free to.

Progress

1. Filed Hir attributes #131808, waiting for review
2. Removed the need for hir attributes from hir_stats.rs remove attribute ids from hir stats (they're simply not needed) #132576
3. Moved attribute related clippy lints to be early pass:
   • Clippy: Move some attribute lints to be early pass (post expansion) #132598
   • Convert disallowed macros to an early pass lint #132645
4. Offline, mostly rewrote rustc_attr

[jieyouxu]

Zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Attribute.20handling.20reworks

[bjorn3]

How will this work with custom tool attributes? These are registered using an attribute inside the crate that uses the attribute on it'w own code.

[jdonszelmann]

hmm, that's a good question. I guess those specifically we couldn't parse and they should stay around as either ast::Attribute or some kind of lowered hir::Attribute a bit like how I describe in the first part of the ideas section. That still makes attributes a lot nicer in the majority of cases.

[jdonszelmann]

Related: #131801

[jdonszelmann]

@rustbot claim