Improved markup, CSS, JS tokenization #1871

karlhorky · 2019-04-26T14:27:39Z

Hi! First, thanks of all for this awesome project! It's affected so many projects around the world. Incredible work!

Motivation

I'm trying to recreate the default VS Code syntax highlighting with Prism.js (via prism-react-renderer) and I'm running into limitations of Prism.js.

Description

I would like to be able to highlight certain tokens that are not granular enough in HTML, CSS and JS.

Would it be something that the Prism.js team is interested in to have more granular tokenization of HTML, CSS and JS? If possible, also with backwards compatibility?

Some first quick examples of things that could be improved across HTML, CSS and JS (compared against VS Code 1.33.1 default dark mode):

HTML: DOCTYPE

HTML: Treating quotes and equals differently than normal punctuation

CSS: URL values

CSS: Arguments in selectors

CSS: Keywords - ex. generic font family names, colors (maybe in CSS-Extra?)

JS: Regex sequences, regex escaping

The code used:

<!DOCTYPE html>

<html lang="en">
  <head>
    <script>
      window.console && console.log('foo');
      function initHighlight(block, cls) {
        try {
          if (cls.search(/\bno\-highlight\b/) != -1)
            return process(block, true, 0x0f) + ` class="${cls}"`;
        } catch (e) {
          /* handle exception */
        }
        for (var i = 0 / 2; i < classes.length; i++) {
          if (checkCondition(classes[i]) === undefined)
            console.log('undefined');
        }
      }
    </script>
    <title>Prism</title>
    <style>
      @font-face {
        src: url(https://lea.verou.me/logo.otf);
        font-family: 'LeaVerou';
      }

      #features li:nth-child(odd),
      footer p {
        font: 100% Consolas, 'Andale Mono', monospace;
        background-image: linear-gradient(
          45deg,
          transparent 34%,
          white 34%,
          white 66%,
          transparent 66%
        );
        text-shadow: 0 1px white;
      }
    </style>
  </head>
</html>

The text was updated successfully, but these errors were encountered:

RunDevelopment · 2019-04-26T16:15:35Z

Thanks for taking the time to make this report!

HTML: DOCTYPE

Agreed.

HTML: Treating quotes and equals differently than normal punctuation

~~I believe that the custom class plugin can be used to add more specific classes to these token which would allow for more granular highlighting.~~
Edit: The plugin can't be used in that way, so we'll have to find a different solution.

CSS: URL values

Agreed. (#1874)

CSS: Arguments in selectors

If you only need even and odd, I'm on board. (#1872)

CSS: Keywords - ex. generic font family names, colors (maybe in CSS-Extra?)

CSS has quite a few keywords, so I don't know if we want to highlight them all because this will increase the size of the CSS language definition. Also, we highlight everything besides the keywords, so wouldn't it be possible to just assume that everything which isn't highlighted is a keyword?

JS: Regex sequences, regex escaping

The Regex language is what you're looking for. This language will add regex tokenization to JS (similar to how CSS-Extras adds new features to CSS).

RunDevelopment · 2019-04-26T20:13:35Z

Regarding the quotes:

I have 2 possible solutions: 1. Add a hook which gives the first = a special class. Change the style of attr-value to look the same as strings and overwrite the color for the = using the special class.

Prism.hooks.add('wrap', function (env) {
	if (env.type === 'punctuation' && env.content === '=') {
		env.classes.push('special-punctuation-name');
	}
});

Not tested but should work. This will also affect the = signs of all other languages but that shouldn't be too much of a problem.

The second solution is to modify the markup language definition:

'attr-value': {
	pattern: /=\s*(?:"[^"]*"|'[^']*'|[^\s'">=]+)/i,
	inside: {
		'punctuation': /^=/,
		'string': {
			pattern: /(\s*)[\s\S]+/,
			lookbehind: true
		}
	}
}

I would careful with modifying the language definition because my other languages rely on markup, so this might cause problems.

RunDevelopment · 2019-04-28T07:52:29Z

Regarding "HTML: Treating quotes and equals differently than normal punctuation":

Another CSS-based solution is to use the fact that the = character will always be the first token inside a attr-value token. So the following to uniquely identify every = character:

.token.attr-value > .token.punctuation:first-child {
  /* styles */
}

karlhorky · 2019-04-28T13:36:41Z

Thanks, these solutions look mostly really good for a start. A couple of points:

CSS has quite a few keywords, so I don't know if we want to highlight them all because this will increase the size of the CSS language definition.

Would it be an acceptable increase to add just the colors and generic font family names for now? If the size is too large, would it be viable to move these out to CSS Extras instead?

Also, we highlight everything besides the keywords, so wouldn't it be possible to just assume that everything which isn't highlighted is a keyword?

If you take a look at the screenshot from VS Code, you'll see that Consolas (and monoospace) are highlighted as plain text, and the other keywords (such as monospace and transparent) are highlighted in the dark orange color. This would not be possible with Consolas, monoospace, monospace and transparent all being treated as plain text.

The Regex language is what you're looking for. This language will add regex tokenization to JS (similar to how CSS-Extras adds new features to CSS).

Ah great, thanks for that, didn't know about that!

karlhorky · 2019-04-28T13:37:24Z

I'll be sure to file any further issues if they come up! (Keeping an eye on pragmatism of course)

RunDevelopment · 2019-05-07T23:06:53Z

Regarding "HTML: DOCTYPE":

Adding tag-highlighting to the doctype token significantly changes the look of Prism's HTML highlighting. It looks really different, so I don't know if we want that.

Instead of changing the language definition for everyone, it might be the simplest solution to delete this token for just you. VS Code treats the doctype like any other tag, AFAIK.

RunDevelopment · 2019-05-07T23:18:02Z

Regarding CSS keywords:

I see the issue with fonts and the like but thought that these minor false positives were acceptable.
Well, all keywords it is then! 😄

karlhorky · 2019-05-08T12:02:03Z

Adding tag-highlighting to the doctype token significantly changes the look of Prism's HTML highlighting. It looks really different, so I don't know if we want that.

Hm... can you elaborate? Since the doctype is already being matched, is there no way of doing some kind of sub-matching on the group already?

Instead of changing the language definition for everyone, it might be the simplest solution to delete this token for just you. VS Code treats the doctype like any other tag, AFAIK.

Do you mean that if I create a custom rule, I can stop Prism from tokenizing the doctype and it will get similar highlighting to the other tags? For example, will ! be treated here as punctuation? Here's VS Code's version again:

Compared to a regular tag, it appears that the tag name and the attribute are similarly highlighted (the only difference is the !):

RunDevelopment · 2019-05-08T15:44:29Z

Here's a little snippet which extends the highlighting of doctypes:

Prism.languages.markup.doctype = {
	pattern: Prism.languages.markup.doctype,
	inside: Prism.util.clone(Prism.languages.markup.tag.inside)
};
Prism.languages.markup.doctype.inside.tag = {
	pattern: /^<!DOCTYPE/i,
	inside: {
		'punctuation': /^<!/
	}
};

The doctype token will then have (almost) the same structure as tag.

If you really don't care about the doctype token, you can also use this snippet:

delete Prism.languages.markup['doctype'];
Prism.languages.markup['tag'].inside['tag'].inside['punctuation'] = /^<[!/]?/;

Both snippets handle the ! and should solve your problem.
I hope that helps.

And with Adding tag-highlighting to the doctype token significantly changes the look of Prism's HTML highlighting, I meant really just that. It looks different.

Before:

After:

karlhorky · 2019-05-08T16:10:31Z

Ah sorry I think there was a misunderstanding here.

And with Adding tag-highlighting to the doctype token significantly changes the look of Prism's HTML highlighting, I meant really just that. It looks different.

Now I'm understanding! Adding highlighting of the doctype is a big change to the visual appearance of HTML files, and it's uncertain if this change is aligned with the goals of the Prism project. Got it!

My understanding was that you meant that the internal implementation looked a lot different (significantly changes the look of Prism's HTML highlighting) and the code became more complex, causing worries about adding that complexity.

Here's a little snippet which extends the highlighting of doctypes
The doctype token will then have (almost) the same structure as tag.

Yeah as long as it's possible with some extension, I'm fine with this route as well.

karlhorky added the enhancement label Apr 26, 2019

This was referenced Apr 26, 2019

Fix token aliases and upgrade Prism FormidableLabs/prism-react-renderer#33

Merged

Improve tokenization FormidableLabs/prism-react-renderer#29

Closed

RunDevelopment added the language-definitions label Apr 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved markup, CSS, JS tokenization #1871

Improved markup, CSS, JS tokenization #1871

karlhorky commented Apr 26, 2019 •

edited

Loading

RunDevelopment commented Apr 26, 2019 •

edited

Loading

RunDevelopment commented Apr 26, 2019

RunDevelopment commented Apr 28, 2019

karlhorky commented Apr 28, 2019

karlhorky commented Apr 28, 2019

RunDevelopment commented May 7, 2019

RunDevelopment commented May 7, 2019

karlhorky commented May 8, 2019 •

edited

Loading

RunDevelopment commented May 8, 2019

karlhorky commented May 8, 2019 •

edited

Loading

Improved markup, CSS, JS tokenization #1871

Improved markup, CSS, JS tokenization #1871

Comments

karlhorky commented Apr 26, 2019 • edited Loading

Motivation

Description

HTML: DOCTYPE

HTML: Treating quotes and equals differently than normal punctuation

CSS: URL values

CSS: Arguments in selectors

CSS: Keywords - ex. generic font family names, colors (maybe in CSS-Extra?)

JS: Regex sequences, regex escaping

RunDevelopment commented Apr 26, 2019 • edited Loading

HTML: DOCTYPE

HTML: Treating quotes and equals differently than normal punctuation

CSS: URL values

CSS: Arguments in selectors

CSS: Keywords - ex. generic font family names, colors (maybe in CSS-Extra?)

JS: Regex sequences, regex escaping

RunDevelopment commented Apr 26, 2019

RunDevelopment commented Apr 28, 2019

karlhorky commented Apr 28, 2019

karlhorky commented Apr 28, 2019

RunDevelopment commented May 7, 2019

RunDevelopment commented May 7, 2019

karlhorky commented May 8, 2019 • edited Loading

RunDevelopment commented May 8, 2019

karlhorky commented May 8, 2019 • edited Loading

karlhorky commented Apr 26, 2019 •

edited

Loading

RunDevelopment commented Apr 26, 2019 •

edited

Loading

karlhorky commented May 8, 2019 •

edited

Loading

karlhorky commented May 8, 2019 •

edited

Loading