Feature: Add solidity and vyper languages

[sambacha]

Languages: Solidity and Vyper

This feature branch adds solidity and vyper, languages used in smart contract programming in
the Ethereum Virtual Machine. It also adds a grammar for ASC X12 EDI (see: https://x12.org).

Bug fix

When running the update grammars command an exception is thrown:

> Done normalizing grammars
> Copying grammars
> Done copying grammars
> Updating source files
internal/fs/utils.js:269
    throw err;
    ^
    Error: ENOENT: no such file or directory, open '/Users/sbacha/shiki/packages/languages/data/asp-vb-net.tmlanguage.json.tmLanguage.json'
    at Object.openSync (fs.js:462:3)
    at Object.readFileSync (fs.js:364:35)
    at /Users/sbacha/shiki/scripts/updateLangSrc.js:44:35
    at Array.map (<anonymous>)
    at Object.<anonymous> (/Users/sbacha/shiki/scripts/updateLangSrc.js:42:4)
    at Module._compile (internal/modules/cjs/loader.js:1015:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1035:10)
    at Module.load (internal/modules/cjs/loader.js:879:32)
    at Function.Module._load (internal/modules/cjs/loader.js:724:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/Users/sbacha/shiki/packages/languages/data/asp-vb-net.tmlanguage.json.tmLanguage.json'

Remediation

$ wget https://raw.githubusercontent.com/microsoft/vscode/master/extensions/vb/syntaxes/asp-vb-net.tmlanguage.json
# asp-vb-net already exists, I did not try and doing `cp asp-vb-net.tmlanguage.json asp-vb-net.tmlanguage.json.tmLanguage.json `
$ mv asp-vb-net.tmlanguage.json.1 asp-vb-net.tmlanguage.json.tmLanguage.json

providing a file named asp-vb-net.tmlanguage.json.tmLanguage.json resolves this issue

Versioning

Running

$ lerna changed
info cli using local version of lerna
lerna notice cli v3.22.1
lerna info Looking for changed packages since v0.2.6
shiki-languages
shiki
shiki-themes
vuepress-plugin-shiki
lerna success found 4 packages ready to publish

Instructions for adding a grammar did not provide any additional information on bumping or otherwise changing versioning/enumeration of the packages.

total changes

running lerna diff shiki-languages we see that:

diff --git a/packages/languages/README.md b/packages/languages/README.md
index 913b1f8..2bcd98a 100644
--- a/packages/languages/README.md
+++ b/packages/languages/README.md
@@ -14,29 +14,44 @@ See [/scripts/pullGrammarsFromGitHub.js](/scripts/pullGrammarsFromGitHub.js)

 ```ts
 export type Lang =
+  | 'JavaScriptReact'
+  | 'MagicPython'
+  | 'MagicRegExp'
+  | 'TypeScriptReact'
   | 'abap'
   | 'actionscript-3'
   | 'ada'
   | 'apex'
   | 'asm'
+  | 'asp-vb-net.tmlanguage.json'
+  | 'asp-vb-net.tmlanguage.json'
   | 'awk'
   | 'bat'
+  | 'batchfile'
   | 'c'
   | 'clojure'
   | 'cobol'
   | 'coffee'
+  | 'coffeescript'
+  | 'cpp.embedded.macro'
   | 'cpp'
   | 'crystal'
   | 'csharp'
+  | 'cshtml'
   | 'css'
   | 'd'
   | 'dart'
   | 'diff'
+  | 'docker'
   | 'dockerfile'
+  | 'edi'
+  | 'electronic-data-interchange'
   | 'elixir'
+  | 'elm-syntax'
   | 'elm'
   | 'erlang'
   | 'fortran'
+  | 'fortran_free-form'
   | 'fsharp'
   | 'git-commit'
   | 'git-rebase'
@@ -49,11 +64,15 @@ export type Lang =
   | 'haskell'
   | 'hcl'
   | 'hlsl'
+  | 'html-derivative'
   | 'html'
+  | 'ignore'
   | 'ini'
   | 'java'
   | 'javascript'
   | 'jinja-html'
+  | 'jsdoc.js.injection'
+  | 'jsdoc.ts.injection'
   | 'json'
   | 'jsonc'
   | 'jsonnet'
@@ -66,19 +85,23 @@ export type Lang =
   | 'log'
   | 'logo'
   | 'lua'
+  | 'make'
   | 'makefile'
   | 'markdown'
   | 'matlab'
   | 'nix'
+  | 'objective-c++'
   | 'objective-c'
   | 'ocaml'
   | 'pascal'
   | 'perl'
   | 'perl6'
   | 'php'
+  | 'platform'
   | 'pls'
   | 'postcss'
   | 'powershell'
+  | 'prolog.swi'
   | 'prolog'
   | 'pug'
   | 'puppet'
@@ -90,24 +113,35 @@ export type Lang =
   | 'rust'
   | 'sas'
   | 'sass'
+  | 'sassdoc'
   | 'scala'
   | 'scheme'
   | 'scss'
+  | 'searchResult'
   | 'shaderlab'
+  | 'shell-unix-bash'
   | 'shellscript'
   | 'smalltalk'
+  | 'solidity'
+  | 'soliditysec'
   | 'sql'
   | 'stylus'
   | 'svelte'
   | 'swift'
   | 'tcl'
   | 'toml'
+  | 'ts'
+  | 'ts'
   | 'tsx'
   | 'typescript'
   | 'vb'
   | 'viml'
+  | 'vue-generated'
+  | 'vue-postcss'
   | 'vue'
+  | 'vyper'
   | 'wasm'
+  | 'wat'
+  | 'wenyan'
   | 'xml'
   | 'xsl'
   | 'yaml'
diff --git a/packages/languages/data/JavaScriptReact.tmLanguage.json b/packages/languages/data/JavaScriptReact.tmLanguage.json
new file mode 100644
index 0000000..92dcef9

are all the changes made to the current languages

Please let me know what corrections should be made, I very much appreciate this work, using tmLanguage for syntax highlighting absolves me of the sin of having to use highlight.js, so thank you for your work. Cheers

[octref]

The only thing I'm worried about is adding niche languages to the base package inflates the size for all users. Is it impossible for you to load the extra grammars through Shiki's API? Would you take it if I create another package shiki-languages-contrib to contain less common languages?

[sambacha]

The only thing I'm worried about is adding niche languages to the base package inflates the size for all users. Is it > impossible for you to load the extra grammars through Shiki's API? Would you take it if I create another package shiki-languages-contrib to contain less common languages?

@octref

That would be great, would actually prefer it that way if possible. Let me know what you would like to do about the bug issue, I can check out the cause of it rather than just covering it up. 

As for the languages, which ones would you consider 'bespoke'? I can update the PR to remove said languages and just include the fixes/languages, or however you would like to proceed, just let me know. Thanks for your time, 

Sam

[octref]

After you rebase to latest master, add the 3 sources in pullGrammarsFromGitHub.js, here's what you'd get:
543cd2d
I think it looks cleaner — not sure how you ended up some irrelevant many changes. Probably due to an outdated pullGHGrammarsFromGitHub that has been fixed upstream.

As for the languages, which ones would you consider 'bespoke'?

I'm not very familiar with Etherum (I should definitely look into it!), but vyper and solidity seems legit. I wouldn't want to include EDI, since all I can find is https://x12.org/examples and nobody seems to be using it 🤷

You can pass to getHighlighter extra options to load grammars. See https://shiki.matsu.io.

Can you open a new PR by only adding vyper/solidity to pullGrammarsFromGitHub.js and run yarn update:grammars? Thanks.

[sambacha]

this is now included in this pr: #102

cheers @octref - thanks!