Feature/change token format

[akinsho]

Following discussions in #1443 to match oni's editor.tokenColors to match with vscode's syntax to allow for cross compatibility with vscode themes this PR changes the theme format to match as well as aims to merge the user, default and theme tokens with priority and as a test adds the editor tokens from vscode/onedark.theme.json

Made a lot of progress here and this PR now uses a TokenScorer to determine which of the available scopes in a list should render for a symbol in a line, this change means that syntax highlighting nearly matches vscode/atom (not accounting for treesitter)

Oni

VSCode

Todo:

• Ensure token merging works as expected.
• Update theme token provider to fill in any necessary gaps as the imported theme does not cover many token scopes which the hover depends on

[codecov]

Codecov Report

Merging #1705 into master will increase coverage by 0.53%.
The diff coverage is 74.52%.

@@            Coverage Diff            @@
##           master   #1705      +/-   ##
=========================================
+ Coverage   44.76%   45.3%   +0.53%     
=========================================
  Files         351     353       +2     
  Lines       14301   14426     +125     
  Branches     1865    1885      +20     
=========================================
+ Hits         6402    6535     +133     
+ Misses       7673    7668       -5     
+ Partials      226     223       -3

Impacted Files	Coverage Δ
browser/src/Services/Themes/ThemeManager.ts	69.73% <ø> (ø)	⬆️
browser/src/UI/components/common.ts	94.52% <ø> (+1.66%)	⬆️
browser/src/Editor/NeovimEditor/markdown.ts	13.58% <0%> (-0.35%)	⬇️
browser/src/neovim/VimHighlights.ts	28.57% <10%> (-46.43%)	⬇️
browser/src/UI/components/QuickInfo.tsx	75.6% <100%> (ø)	⬆️
...es/SyntaxHighlighting/SyntaxHighlightReconciler.ts	79.54% <100%> (+3.45%)	⬆️
browser/src/UI/components/QuickInfoContainer.tsx	33.33% <33.33%> (+2.56%)	⬆️
browser/src/Services/TokenColors.ts	57.14% <41.66%> (-12.86%)	⬇️
browser/src/neovim/NeovimTokenColorSynchronizer.ts	78.84% <64.7%> (-5.94%)	⬇️
...Services/SyntaxHighlighting/TokenThemeProvider.tsx	90.62% <94%> (+67.22%)	⬆️
... and 26 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4bdbdc5...0287851. Read the comment docs.

[akinsho]

Granted tests pass, I believe this is done (for now... from reading around there are loads of tweaks that could be made to improve how we highlight in the long run).
Would appreciate a review (keep in mind the change will only be noticeable if using onedark)

-> side note for my future self or anyone who wants to look at it in markdown.ts, where symbols are defined we dont apply any ranking logic but instead pass the all matching tokens to each symbols so all symbols have several applicable class names -> starting point for optimising the hover box at least

[CrossR]

I've given this a quick look over, but is there anything specific you want a second look at?

[akinsho]

@CrossR I think my main goal was to avoid breaking any of the syntax highlighting whilst allowing for the use of editor tokens from vscode themes, so that we can copy their editor tokens into our oni themes and they should match the highlighting (almost*) means with this PR anyone making a port of a theme can get really good highlighting by borrowing the tokens.

• there are quite a few quirks with how the highlighting is applied which I find hard to quantify prior to and despite this PR (I didnt alter the logic) it still occasionally disappears, or is partially applied, on sourcing vimrc as one example, one change I did make was converting the hi oni_highlight_x guifg={ourColor} to have a hi! which I believe forces the highlight

[CrossR]

Everything looked good and didn't seem to have broken anything, and the code changes look sensible.

Perhaps its worth sticking a disclaimer in the insiders chat on discord, just so people can be aware and maybe they'll spot something we didn't, since as you've said a lot of the highlighting stuff seems to change very very randomly.

Excited for this, the highlighting looks much nicer!

[akinsho]

Will post a little message on discord anyone who uses onedark should see the new tokens and maybe might notice if anything is off although I think the reason for the bugs, although not at all sure, is that the highlighting is sort of cached and or rather we save the lines we have highlighted but I'm not sure if that's enough for vim as there are a lot of things that could alter or clear highlighting in vim and if we believe the line is highlighted then it won't reapply it (or at least I think this is a possible explanation for why sourcing your vimrc will clear all textmate highlighting)