Alternative ways of implementing Javascript Plugins

[leumasme]

The current implementation for NodeJS plugins is frankly not good. Communication is via ""JSON RPC"" via launch arguments and stdout (console.log), which means that the Node process will have to be restarted for each plugin call (-> each single-character change of the search query, etc)
Although it was discussed in the Discord that this could/would be optimized by using stdin which would mean no more process restarting, it feels a bit like you're reinventing the wheel here.
Although awesome people like @Joehoel and @gabrielcarloto
have already created templates and libraries to enhance the plugin dev experience, the overall situation is still unsatisfactory imo.

There are existing solutions like JeringTech/Javascript.NodeJS which seem very promising from a performance aspect. The linked project allows reusing a single NodeJS process even for different scripts, which could massively reduce overhead when many JS plugins are in use.
Running the example from a string literal yields ~700ms on the first script run (which has to start the node process) and ~2ms for consecutive script runs.
Further performance gains for (large) scripts are to be expected when loading them from a file instead of a string literal:
When using InvokeFromFileAsync, NodeJS always caches the module using the .js file's absolute path as cache identifier. This is great for performance, since the file will not be re-read or recompiled on subsequent invocations.
I'm not sure how the cold startup time compares to the current execution time for a NodeJS plugin, more stats on this are TODO.
A possible downside from using this library is that it currently doesn't seem to support returning multiple results at different times (which the current approach allows) - i created JeringTech/Javascript.NodeJS#158 regarding this.

For now, this thread shall serve as a discussion point for possible better solutions to implement JS Plugins.
Any other ideas on this are very welcome.

[leumasme]

According to a reply in the issue I opened, the library is indeed currently not supporting two-way communication although it has been requested before.
Some internals on how the Node side of the lib seems to work:
It spawns a node process with a predefined script that opens a HTTP server. The .NET library then sends Scripts or cache keys,function names and arguments to the HTTP server, where the node script loads the transmitted script via the "module" NodeJS api. Loaded scripts are cached this way, but by the design of HTTP, each request can only get one response.

Further questions to be asked:
Are there any plugins that (could) even utilize this (returning multiple results at different times)?
Is forking the lib and adding this functionality via websockets viable / a good trade-off vs 'basic' JSON RPC V2 as it's currently planned (time spent vs gained Performance/cleanness)? Maybe instead, adding JSON RPC V2 as it was planned (how far is the current progress on that?) and then (later?) improving it to use a single Node process and cache modules is a better idea?

@taooceros writing down your opinions here may be a good idea so that they don't get lost in the discord.
You said that yeah I think we can implement a bidirectional communication single process app - did you mean re-implementing the structure of a node process with a predefined script from scratch, just with a different protocol to allow 2-way communication? Otherwise I'm not sure what you wanted to say.