Crash in encoding rpc message (exception too much recursion)

[MichelSc]

This crash was observed with versions of theia: 1.34.0, 1.34.4, 1.37.2, 1.38.0.

Theia is built with msgpackr version 1.9.2, which agrees with the version required with @theia/core, namely version "^1.6.2".

While opening a file with build-in opener (*.txt, *.json, *sparql, ...) I get in the front-end an exception Uncaught (in promise) Error: Error during encoding: 'too much recursion'" and as from this time, the editor for that file does not work correctly.`

Reproduction scenario: open an empty text file.

The exception is thrown from the call to encode( ) at line

theia/packages/core/src/common/message-rpc/rpc-message-encoder.ts

Line 148 in 0caea2c

buf.writeBytes(this.msgPack.encode(value));

The msgPack.encode gets in an infinite recursion, with a cycle of length 1, at the line
https://github.com/kriszyp/msgpackr/blob/30875db0098d2fc8554460f62b9012b1ad6f468f/pack.js#L492

because for the passed object value, method toJSON() returns this (so it calls itself with the received value as parameter).

The passed value is a Selection, that inherits from a Range, which implement toJSON() as return this (see https://github.com/microsoft/vscode/blob/23850e23637ad81e479fc455d3e90789cc0404d4/src/vs/editor/common/core/range.ts#L493)

This explains the infinite recursion.

If we downgrade msgpackr to 1.8.5, the issue disappears (msgpackr does not use toJSON().

(but not sure msgpackr is the culprit; Range.toJSON looks suspect to me).

Work around: add in your package.json at project level a resolution rule

  "resolutions": {
    "**/@theia/core/msgpackr": "<=1.8.5"
  },

[msujew]

@MichelSc We are aware of this issue (see #12526) and provide a fix for users in the documentation (see #12527). It's similar to what you outline.