Support for FontStyle.Strikethrough (tracking issue)

[ghost]

Just a note for anyone looking into this: for text grammars, it seems that adding an extra font-style will not play nicely with the current binary format we use for tokens

Such a change needs to begin in the project vscode-textmate. The bit encoding format for tokens is defined in that project at:

vscode-textmate/src/main.ts

Lines 201 to 233 in a53ae3c

	/**
	* Helpers to manage the "collapsed" metadata of an entire StackElement stack.
	* The following assumptions have been made:
	* - languageId < 256 => needs 8 bits
	* - unique color count < 512 => needs 9 bits
	*
	* The binary format is:
	* - -------------------------------------------
	* 3322 2222 2222 1111 1111 1100 0000 0000
	* 1098 7654 3210 9876 5432 1098 7654 3210
	* - -------------------------------------------
	* xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx
	* bbbb bbbb bfff ffff ffFF FTTT LLLL LLLL
	* - -------------------------------------------
	* - L = LanguageId (8 bits)
	* - T = StandardTokenType (3 bits)
	* - F = FontStyle (3 bits)
	* - f = foreground color (9 bits)
	* - b = background color (9 bits)
	*/
	export const enum MetadataConsts {
	LANGUAGEID_MASK = 0b00000000000000000000000011111111,
	TOKEN_TYPE_MASK = 0b00000000000000000000011100000000,
	FONT_STYLE_MASK = 0b00000000000000000011100000000000,
	FOREGROUND_MASK = 0b00000000011111111100000000000000,
	BACKGROUND_MASK = 0b11111111100000000000000000000000,
	LANGUAGEID_OFFSET = 0,
	TOKEN_TYPE_OFFSET = 8,
	FONT_STYLE_OFFSET = 11,
	FOREGROUND_OFFSET = 14,
	BACKGROUND_OFFSET = 23
	}

Adding the strikethrough font style would require four bits instead of three

Once that project has a new token bit encoding format, it needs to be published with a new major version (since this is a breaking change) and then adopted in VS Code.

Also, the enums are powers of 2 because they represent bits, so FontStyle.Strikethrough would be 8. But for it to be 8, then all the other information bits need to be shifted by 1.

[alexdima]

To make room for it, I think StandardTokenType could be shrinked to only 2 bits (it has only 4 possible values: Other, Comment, String, RegEx and they don't need to be masks).

[ghost]

Awesome! Normally I would start work on this immediately but I have to port my Textmate LSP to the browser and its a rather involved task because of the moving parts. I could get to this in the New Year? 🤔

[ghost]

Draft PR at #159