Syntax highlighting - needs automated tests

[ivanz]

The syntax highlighting is based on a regex-based grammar and because there aren't any automated tests to validate changes - it is easy to cause regressions without knowing.

[DustinCampbell]

We're planning to ship 1.4 soon. Once that happens, I'll start working on this. PR #725 integrates mocha for testing.

[ivanz]

I literally started looking into this 10 minutes ago and was just in the process of setting up typings and mocha. I was looking at vscode-textmate and if that can be used to check the token output. :)

[ivanz]

@DustinCampbell I have built something that works well for me, is quite minimal on top of the vscode-textmate and looks like below in practice.

I would create a pull request now, but there will be some plumbing duplication with #725 (mocha, chai and typings set-up). I will wait for once #725 lands in and then create one on top.

import { assert } from 'chai'
import { Tokenizer, Token, Tokens } from './tokenizer'

let input = `
namespace TestNamespace
{
}`;

let tokens: Token[] = new Tokenizer().tokenize(input);

assert.include(Tokens.NamespaceKeyword("namespace"), tokens);
assert.include(Tokens.NamespaceIdentifier("TestNamespace"), tokens);

[DustinCampbell]

Cool! Looking forward to it!

[ivanz]

@DustinCampbell I have created a temporary repo which captures my current approach here: https://github.com/ivanz/omnisharp-csharp-grammar-test-harness in order to get early feedback. Let me know what your thougths are!

Example tests for "namespace"are here and the interesting implemntation bits are here. You can build and run with npm test if you end up having a play.

The idea is to wrap vscode-textmate in an easy to consume interface and expose an easy way to check if tokens were created correctly without hardcoding token ids in tests, etc.

[ivanz]

I've migrated the above code into a pull request in #742