Create test framework for grammar rules

[vinistock]

Currently, there are no tests for our grammar files. Not having tests is always bad, but it's especially worse for this case since the Ruby grammar file is immense and filled with complex regexes that try to understand Ruby's ambiguous syntax.

Let's try to design a little test framework that allows us to assert the token types for a string of Ruby code. The framework should apply the grammar rules on the given source string and then somehow allow us to assert the tokens based on positions.

Something like this:

test("Class declaration tokens", () => {
  const tokenTypes = applyGrammar("class Foo < Bar; end");
  assert.strictEqual(tokenTypes[0].type, "keyword");
  assert.strictEqual(tokenTypes[1].type, "class.declaration");
});

[borama]

FWIW, I like the https://github.com/PanAeon/vscode-tmgrammar-test approach and I've had great success with building a test suite for my VS Code Slim extension using it. I may even try to start building some tests if there is interest.

[andyw8]

I hadn't realised the extension already has something for testing grammar rules:

#2493

[vinistock]

Indeed, I forgot we already have a setup for testing grammar rules. I'll close this issue then.