Expand monarch functionality to allow state access within rules

[jeremy-rifkin]

I think monarch is not powerful enough to tokenize constructs such as C++ raw string literals in the general case. This is an attempt to expand monarch functionality so it can.

Monaco tokenization of C++ raw strings is broken when the d-char-sequence  contains double quotes microsoft/monaco-editor#3128

This is because https://github.com/microsoft/monaco-editor/blob/main/src/basic-languages/cpp/cpp.ts looks for the end of a raw string literal by searching for /(.*)(\))(?:([^ ()\\\t"]*))(\")/ and then checks $3==$S2 to see if the end has in fact been found, but only compares what is between a closing parenthesis and the first double quote.

As far as I can tell, there is no way to solve this currently in monarch, without doing something hacky like hard-coding for all d-char-sequences of 16 or fewer characters, for example (the C++ standard does specify this limit). Maybe there is a way that I haven't thought of, in which case this PR isn't needed. Most languages do not seem to place such limitations on the delimiter sequences, e.g. PHP and C#. Rust tokenization can be improved as well with the tools provided by the PR.

---

This PR expands monarch functionality by allowing $S2 to be used within rule regexes, so the fix for microsoft/monaco-editor#3128 can be something along the lines of

        root: [
            // C++ 11 Raw String
            [/@encoding?R\"(?:([^ ()\\\t]*))\(/, { token: 'string.raw.begin', next: '@raw.$1' }],
            ...
        ],
        raw: [
+           [/.*\)$S2\"/, 'string.raw', '@pop']
-           [
-               /(.*)(\))(?:([^ ()\\\t"]*))(\")/,
-               {
-                   cases: {
-                       '$3==$S2': [
-                           'string.raw',
-                           'string.raw.end',
-                           'string.raw.end',
-                           { token: 'string.raw.end', next: '@pop' }
-                       ],
-                       '@default': ['string.raw', 'string.raw', 'string.raw', 'string.raw']
-                   }
-               }
-           ],
            [/.*/, 'string.raw']
        ],

I have not fully tested this PR yet.

[jeremy-rifkin]

@microsoft-github-policy-service agree

[jeremy-rifkin]

The automated test cases and unit tests pass. I could use some guidance on how to test the changes in conjunction with monaco / a test monarch grammar.

[alexdima]

Thank you!