tmbundle: Make tmLanguage compatible with PCRE2

[lildude]

👋 I'm the lead maintainer of the https://github.com/github/linguist library which is used for language detection and providing the syntax highlighting for languages on GitHub.com, and we use this grammar.

Our grammar compiler has found several problems with your grammar which I thought I'd let you know about.

These regexes have quite a few problems as you can see in the regex101 link after each:

marko-tmbundle/syntaxes/marko.tmLanguage.json

Line 779 in 60ded48

"end": "(?=[,;\\](]|/>|(?<=[^=])>|(?<!(?:^|[!~*%&^|?:]|[!~*%&^|?/<>+=-]=|=>|>{2,}|[^.]\\.|[^-]-|^\\s*\\+\\+|[^\\+]\\+{2}*\\+|[a-zA-Z0-9%).<\\]}]\\s*/|\\b(?<![.]\\s*)(?:await|async|class|function|keyof|new|typeof|void))\\s*)(?:\\n|[ \\t]+(?![\\n{+!~*%&^|?:]|[<>/=-]=|=>|>{2,}|\\.[^.]|-[^-]|/[^>]|(?:in|instanceof|as|extends)\\s+[^:=/,;>])))",

https://regex101.com/r/pSG73T/1

marko-tmbundle/syntaxes/marko.tmLanguage.json

Line 108 in 60ded48

"end": "(?=[,;\\]]|/>|(?<=[^=])>|(?<!(?:^|[!~*%&^|?:]|[!~*%&^|?/<>+=-]=|=>|>{2,}|[^.]\\.|[^-]-|^\\s*\\+\\+|[^\\+]\\+{2}*\\+|[a-zA-Z0-9%).<\\]}]\\s*/|\\b(?<![.]\\s*)(?:await|async|class|function|keyof|new|typeof|void))\\s*)(?:\\n|[ \\t]+(?![\\n{(+!~*%&^|?:]|[<>/=-]=|=>|>{2,}|\\.[^.]|-[^-]|/[^>]|(?:in|instanceof|as|extends)\\s+[^:=/,;>])))",

... and repeated again at:

marko-tmbundle/syntaxes/marko.tmLanguage.json

Line 130 in 60ded48

"end": "(?=[,;\\]]|/>|(?<=[^=])>|(?<!(?:^|[!~*%&^|?:]|[!~*%&^|?/<>+=-]=|=>|>{2,}|[^.]\\.|[^-]-|^\\s*\\+\\+|[^\\+]\\+{2}*\\+|[a-zA-Z0-9%).<\\]}]\\s*/|\\b(?<![.]\\s*)(?:await|async|class|function|keyof|new|typeof|void))\\s*)(?:\\n|[ \\t]+(?![\\n{(+!~*%&^|?:]|[<>/=-]=|=>|>{2,}|\\.[^.]|-[^-]|/[^>]|(?:in|instanceof|as|extends)\\s+[^:=/,;>])))",

https://regex101.com/r/NlVs41/1

These are the errors our compiler reported:

• Invalid regex in grammar: text.marko (in syntaxes/marko.tmLanguage.json) contains a malformed regex (regex "(?=[,;\](]|/>|(?<=[^=])>|(?<!(?:...": nothing to repeat (at offset 105))
• Invalid regex in grammar: text.marko (in syntaxes/marko.tmLanguage.json) contains a malformed regex (regex "(?=[,;\]]|/>|(?<=[^=])>|(?<!(?:^...": nothing to repeat (at offset 104))
• Invalid regex in grammar: text.marko (in syntaxes/marko.tmLanguage.json) contains a malformed regex (regex "(?=[,;\]]|/>|(?<=[^=])>|(?<!(?:^...": nothing to repeat (at offset 104))

[DylanPiercey]

Will look shortly. Thanks for the report!

[DylanPiercey]

@lildude from what I can tell the regex functions correctly. The validator you are using is using PCRE2 however my understanding was that the regex's in tmgrammars are intended to be handled by oniguruma. Is this not the case for linguist?

[lildude]

Is this not the case for linguist?

No. GitHub uses PCRE for grammar parsing for performance reasons.