Accept additional user-defined syntax classes in fenced code blocks

[GuillaumeGomez]

Part of #79483.

This is a re-opening of #79454 after a big update/cleanup. I also converted the syntax to pandoc as suggested by @notriddle: the idea is to be as compatible as possible with the existing instead of having our own syntax.

Motivation

From the original issue: #78917

The technique used by inline-c-rs can be ported to other languages. It's just super fun to see C code inside Rust documentation that is also tested by cargo doc. I'm sure this technique can be used by other languages in the future.

Having custom CSS classes for syntax highlighting will allow tools like highlight.js to be used in order to provide highlighting for languages other than Rust while not increasing technical burden on rustdoc.

What is the feature about?

In short, this PR changes two things, both related to codeblocks in doc comments in Rust documentation:

• Allow to disable generation of language-* CSS classes with the custom attribute.
• Add your own CSS classes to a code block so that you can use other tools to highlight them.

The custom attribute

Let's start with the new custom attribute: it will disable the generation of the language-* CSS class on the generated HTML code block. For example:

/// ```custom,c
/// int main(void) {
///     return 0;
/// }
/// ```

The generated HTML code block will not have class="language-c" because the custom attribute has been set. The custom attribute becomes especially useful with the other thing added by this feature: adding your own CSS classes.

Adding your own CSS classes

The second part of this feature is to allow users to add CSS classes themselves so that they can then add a JS library which will do it (like highlight.js or prism.js), allowing to support highlighting for other languages than Rust without increasing burden on rustdoc. To disable the automatic language-* CSS class generation, you need to use the custom attribute as well.

This allow users to write the following:

/// Some code block with `{class=language-c}` as the language string.
///
/// ```custom,{class=language-c}
/// int main(void) {
///     return 0;
/// }
/// ```
fn main() {}

This will notably produce the following HTML:

<pre class="language-c">
int main(void) {
    return 0;
}</pre>

Instead of:

<pre class="rust rust-example-rendered">
<span class="ident">int</span> <span class="ident">main</span>(<span class="ident">void</span>) {
    <span class="kw">return</span> <span class="number">0</span>;
}
</pre>

To be noted, we could have written {.language-c} to achieve the same result. . and class= have the same effect.

One last syntax point: content between parens ((like this)) is now considered as comment and is not taken into account at all.

In addition to this, I added an unknown field into LangString (the parsed code block "attribute") because of cases like this:

/// ```custom,class:language-c
/// main;
/// ```
pub fn foo() {}

Without this unknown field, it would generate in the DOM: <pre class="language-class:language-c language-c">, which is quite bad. So instead, it now stores all unknown tags into the unknown field and use the first one as "language". So in this case, since there is no unknown tag, it'll simply generate <pre class="language-c">. I added tests to cover this.

Finally, I added a parser for the codeblock attributes to make it much easier to maintain. It'll be pretty easy to extend.

As to why this syntax for adding attributes was picked: it's Pandoc's syntax. Even if it seems clunkier in some cases, it's extensible, and most third-party Markdown renderers are smart enough to ignore Pandoc's brace-delimited attributes (from this comment).

Raised concerns

It's not obvious when the language-* attribute generation will be added or not.

It is added by default. If you want to disable it, you will need to use the custom attribute.

Why not using HTML in markdown directly then?

Code examples in most languages are likely to contain <, >, & and " characters. These characters require escaping when written inside the <pre> element. Using the ``` code blocks allows rustdoc to take care of escaping, which means doc authors can paste code samples directly without manually converting them to HTML.

cc @poliorcetics
r? @notriddle

[GuillaumeGomez]

Fixed the feature version errors.

[notriddle]

I'm surprised that it's not aligning the syntax more closely with RFC 3311.

Last time, on that RFC, I did a Babelmark run to try to design a syntax that would produce reasonable results in most Markdown parsers. The important thing is that this syntax should gracefully degrade in mdBook, docs.rs, crates.io, GitHub, and other widely-used Markdown renderers, so that people can use #[doc=include!("README.md")] and easily write documentation that works in both Rustdoc and in whatever other renderer they're using.

p.s. I still think we should align Rustdoc's handling of code strings as closely as possible with Pandoc's. Even if it seems clunkier in some cases, it's extensible, and most third-party Markdown renderers are smart enough to ignore Pandoc's brace-delimited attributes.

[notriddle]

Babelmark run for the proposed syntax shows that this syntax behaves badly in other markdown parsers.

[GuillaumeGomez]

I'm surprised that it's not aligning the syntax more closely with RFC 3311.

I wasn't aware of it. Using = sounds good to me as well.

About following pandoc, it depends on one big thing: do we want to allow code-blocks to have user-defined IDs? Also, as you mentioned, it's a bit clunkier. class= is easier to read and doesn't require to implement a new parser for it. And finally, do we want it to be extensible?

EDIT: In any case, both solutions sound good to me. But I really find the pandoc one ugly. ^^'

EDIT2: Thinking about it some more, using an existing syntax would actually be wiser so I think going for the pandoc one is the right move here. I'll update the PR for it and specifically reject id for the time being.

Note for myself: should only allow for class=something and .something. Any other attribute will be ignored.

[notriddle]

And finally, do we want it to be extensible?

We've already got two features that want key-value pairs here (RFC 3311, and this one). It seems pretty arbitrary to think there won't be more.

In any case, both solutions sound good to me. But I really find the pandoc one ugly.

I know their documentation always shows key=value pairs with quoted values, but you don't actually need to quote them. Here's an example where rust {class=test} does what you'd expect.

[GuillaumeGomez]

I switched to the pandoc syntax as suggested and created a new parser for the markdown codeblock attributes. Any attribute which is not a class (so either . or class=) will emit a warning inside "attribute block" (ie {}).

[notriddle]

Needs updated.

[GuillaumeGomez]

I added the suggested tests, I updated the documentation and I added the support for double quotes (which is much more permissive than pandoc's).

[notriddle]

I'd like to create an eBNF to describe this grammar.

There's a few aspects of it, like the way quotes and dots are handled, that make writing one difficult. Would you be okay with making this syntax a lot stricter in order to make it easier to write a spec?

[GuillaumeGomez]

I'm open to it. The current handling could have an eBNF as well I think. But you seem like you have an idea for a syntax so don't hesitate to suggest an eBNF. :)

[notriddle]

@GuillaumeGomez

Alright, I've put together an eBNF that seems to "fit the spirit of this change," by passing many of the test cases and generally acting "reasonable." It, and a combine model of it, can be found at https://gitlab.com/notriddle/fuzz-fenced-code-blocks.

[notriddle]

Some noticeable differences between the two of them:

• It doesn't allow opening quotes in the middle of tokens. {key=v"alue"} is invalid
• It doesn't allow barewords to contain most ASCII punctuation. You have to quote them. For example, .foo=bar is a particularly nasty and ambiguous parse, which is solved by rejecting it as invalid

Some notable similarities:

• It allows keys in "key"=value attributes to be quoted
• It allows lang tokens and dotted classes to be quoted, like "rust" {."myClass"}
• It allows multiple attribute lists, and they can be reordered, like {.myClass} rust {class=myClass2}
• Commas are allowed as separators

All of this stuff should be changeable, if any of it is a problem (though none of it seems too onerous to me).

[bors]

☔ The latest upstream changes (presumably #111001) made this pull request unmergeable. Please resolve the merge conflicts.

[RalfJung]

Is it possible that this caused a regression? We're seeing rustdoc errors about "custom classes" on nightly Miri CI since this morning. See Zulip.

[GuillaumeGomez]

Yes, it should emit a warning and not an error. I'll fix this right away.