Wasm Plugin System: Pre-RFC Discussion

[kirawi]

Before we can fully commit to implementing the Wasm plugin system, I believe it is necessary that we first engage in discussion around what we want out of it, and how we believe it can best be implemented. Discussion here should eventually lead to an RFC, and then to resuming work on the Wasm plugins PR (#455). Discussion of alternate runtimes (e.g. Wasmer) is out-of-scope and can be discussed at a later date.

Introduction to Wasm: https://developer.mozilla.org/en-US/docs/WebAssembly/Concepts
Introduction to witx: https://github.com/WebAssembly/WASI/blob/main/docs/witx.md
Introduction to Wasmtime: https://docs.wasmtime.dev/

[dsseng]

Maybe plugins could live in separate plugin host processes. This will allow them to compute pretty heavy things without making main editor UI process slow to respond to user's interaction.

[kirawi]

I'm not sure about whether that would be desirable: https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.html:

One argument for process separation is to improve overall system reliability. For example, Chrome has a process per tab, and if the process crashes, all you get is an “Aw, snap” without bringing the whole browser down. I think it’s worth asking the question: is it useful to have the front-end continue after the core crashes, or the other way around? I think probably not; in the latter case it might be able to safely save the file, but you can also do that by frequently checkpointing.

We're discussing this feature in a different context, however, and it's worth asking that question ourselves. In my opinion, I think that it would be a good feature to have in the long-term, but it could add significant complexity. I guess we could investigate this after the MVP.

[archseer]

I agree here, we should stick with treating plugins as simple FFI calls for now. While I think running these plugins with their own stateful loops will be useful, we can incrementally get there. WASM also brings enough separation that we shouldn't need to do a whole browser-like process per plugin.

[dsseng]

Yes, since multiprocessing will require IPC (at least stdio) for communicating between threads. Probably we want to get exact answers on what happens if WASM thread falls into infinite loop (that's probably the only issue so far, others will return an error). I haven't looked deep into Wasmtime, but looks like this code will freeze as well:

plugin = wasm.load("plugin.wasm");
status = plugin.get_status(); // freezes because of plugin bug
// ... main process freezed and cannot proceed

[archseer]

We can both set a stack_limit to prevent infinite loops, as well as use an interrupt handle to timeout execution.

[dsseng]

Yes, then for now everything can run in main thread.

[kirawi]

Is there any reason that we would want to expose an API for helix-term? I think it makes sense to group up all the APIs for the rest of the crates inside helix-plugin which would depend upon helix-view at most. but if helix-term is thrown into the equation I'm not sure.

I'll try to spend some more time digging into how the bindings work to wrap my head around it.

[sudormrfbin]

At the moment helix-term has some code that is supposed to be in helix-view, like commands, and I think the plugin crate will also have to depend on helix-lsp for LSP integration and helix-syntax for treesitter integration.

[kirawi]

helix-syntax only contains the shared library loading for tree-sitter irc. You're probably thinking of helix-core/src/syntax.rs.

[sudormrfbin]

Ah yeah my bad, then we'd effectively only need lsp and view.

[Th3Whit3Wolf]

I'm not sure if this has been discussed somewhere else and I apologize if it has already been discussed but I think it would be nice if plugins came with a config file, that helix could read and understand what commands, keybindings, views, plugin settings, when to activate a plugin (onLanguage, onCommand, onDebug, workspaceContains, onFileSystem, onView, onUri,onStartupFinished), and other plugin information.

Vscode requires extension declare this in their package.json and can also use the data to display menu for their extension with descriptions and help tooltips on hover. Vscode's API I think is probably worth looking at in general since it's designed for many different kinds of plugins with different feature set and is pretty mature at this point. I think this kind of api encourages good documentation, having possible plugin configuration settings predefined allows for type checking/linting of the user configuration settings, and will give any helix gui the ability to more easily render the plugin settings in an intuitive way as well as other aspects of the plugin.

Below are some examples of json settings and how vscode renders them (all examples are taken from vsocde's api reference site ).

{
  "contributes": {
    "commands": [
      {
        "command": "extension.sayHello",
        "title": "Hello World",
        "category": "Hello",
      }
    ]
  }
}

turns into

{
  "contributes": {
    "configuration": {
      "title": "TypeScript",
      "properties": {
        "typescript.useCodeSnippetsOnMethodSuggest": {
          "type": "boolean",
          "default": false,
          "description": "Complete functions with their parameter signature."
        },
        "typescript.tsdk": {
          "type": ["string", "null"],
          "default": null,
          "description": "Specifies the folder path containing the tsserver and lib*.d.ts files to use."
        }
      }
    }
  }
}

turns into

{
  "gitMagic.blame.heatmap.location": {
    "type": "string",
    "default": "right",
    "enum": ["left", "right"],
    "enumDescriptions": [
      "Adds a heatmap indicator on the left edge of the gutter blame annotations",
      "Adds a heatmap indicator on the right edge of the gutter blame annotations"
    ]
  }
}

turns into

{
  "contributes": {
    "keybindings": [
      {
        "command": "extension.sayHello",
        "key": "ctrl+f1",
        "mac": "cmd+f1",
        "when": "editorTextFocus"
      }
    ]
  }
}

turns into (notice the Ctrl+F1 in top right corner of prompt)

[Th3Whit3Wolf]

It's hard to detect X to be a python project

Maybe something could be baked for basic project root protect like vim-rooter, for example check if in a git repository and/or have some predefined workspaces for languages with well defined workspace structures (Rust workspaces always have a Cargo.toml in root directory for instance). Walking through directories in parallel and regexing file names can be done quickly and is only necessary at startup (could also be asynchronous to not add any delay to startup).

• This could also be done to simultaneously load environment variables from .env's

After querying workspace a plugin could either match on workspace or as you mentioned use a regex to determine whether or not it should be loaded.

[kirawi]

Perhaps we could use a plugin.toml configuration file to specify the permissions of a plugin upfront? It otherwise feels unnecessary to specify tooltips and commands in the config (we can probably generate those on the fly).

[Th3Whit3Wolf]

@kirawi

You are right, we definitely could have commands be registered through the wasm runtime on startup. Personally it feels better to have static content declared in a configuration file rather than have the plugins register their command and the commands properties, but either would work.

Having the the commands predefined also allows the user's config to be linted and could warn the user that their keybindings are to an invalid command.

Menus and tooltips would be generated by descriptions of a plugins setting and not something that would have to be specified. Here is an example

"rust-analyzer.checkOnSave.command": {
    "markdownDescription": "Cargo command to use for `cargo check`.",
    "default": "check",
    "type": "string"
}

Would create a menu like this

Rust-analyzer > Check On Save: Command

"Cargo command to use for cargo check.

with a text box underneath it.

"rust-analyzer.updates.channel": {
    "type": "string",
    "enum": [
        "stable",
        "nightly"
    ],
    "default": "stable",
    "markdownEnumDescriptions": [
        "`stable` updates are shipped weekly, they don't contain cutting-edge features from VSCode proposed APIs but have less bugs in general.",
        "`nightly` updates are shipped daily (extension updates automatically by downloading artifacts directly from GitHub), they contain cutting-edge features and latest bug fixes. These releases help us get your feedback very quickly and speed up rust-analyzer development **drastically**."
    ],
    "markdownDescription": "Choose `nightly` updates to get the latest features and bug fixes every day. While `stable` releases occur weekly and don't contain cutting-edge features from VSCode proposed APIs."
}

Will create a menu like this

Rust-analyzer > Updates: Channel

Choose nightly updates to get the latest features and bug fixes every day. While stable releases occur weekly and don't contain cutting-edge features from VSCode proposed APIs.

With a dropdown box where you can choose stable or nightly and another box below it (or above if there isn't enough space in the window to render below) that will display

stable updates are shipped weekly, they don't contain cutting-edge features from VSCode proposed APIs but have less bugs in general.

or

Choose nightly updates to get the latest features and bug fixes every day. While stable releases occur weekly and don't contain cutting-edge features from VSCode proposed APIs.

depending on which item the cursor is on.

Not sure why I mentioned tooltips, the only tooltip vscode shows is whatever your mouse cursor hovers over, it's in now way helpful.

[Th3Whit3Wolf]

@kirawi I update my original post with images and configuration settings taken from vscode's api documentation website.

[kirawi]

Yeah, that makes a lot more sense than my original idea of using a register_commands callback. I have no idea how that went over my head, haha.

[kirawi]

For the past week, I've been trying to find the best way to architect the runtime, and I've started to feel that we could take a lot of inspiration from Bevy's ECS and how it's architected. More specifically, I feel that their Systems scheduling is exactly what we want.

[kirawi]

On Matrix, this was the initial approach we discussed:
Plugins can implement a register_commands() function, and this function will be executed across all plugins on startup. The function defines a Command, like we already do for the existing commands, except the callback will refer to a user-defined function inside the plugin. When the command is invoked, the callback will be called and be passed a context (whether this be the already existing Context or a new wrapper type entirely) that implements the associated API traits for the plugin to use.

[raphCode]

Sounds nice, but is code execution possible without issued Commands? For example a timeout which changes back to normal mode after inactivity? In my understanding, this requires grabbing all key events to reset the timeout and running code asynchronous outside of a callback when the timers expires.
I bet additional flexibility over just registering some keyboard Commands enables even more cool applications...

[kirawi]

Possibly. Commands would probably need to be reworked for such a thing, but such plugins are planned post-MVP.

[ym-han]

Just re-posting a big-picture question of mine from another github issue: Is it going to be at all possible to somehow use or adapt extensions from other editors (either from neovim or emacs or vscode), on any of the approaches that are being discussed? I haven't really seen any mention of this point before, but I think there's something to be said about seeing if there's any way to capitalize on all the incredible work people have already put into plugins for other editors and ecosystems, if possible. (This is the path OniVim2 took --- native vim but with the capability to use vscode extensions.)

I don't think this is a decisive consideration, but it does seem worth considering.

[kirawi]

I think that supporting Vim/Neovim/VSCode/etc. plugins would be very difficult to attempt because of the differing philosophies and features of each editor. OniVim2 is somewhat successful because it is attempting to match VSCode as much as possible, but this goes against one of Helix's primary goals: "Don't try to be everything for everyone. There are many great editors out there to choose from. Let's make Helix one of those great options, with its own take on things."

We may or may not have a similar API to the aforementioned editors, but directly supporting their plugin ecosystems is a non-goal. What I can see happening is a plugin being developed that can process vsx extensions.

[ym-han]

I hadn't realized it might be possible to dev a plugin that can process vsx extensions. That would be very cool, though I imagine it'd be quite hard to make one that would be able to process arbitrary vsx extensions.

[kirawi]

@CBenoit and I talked a couple of days ago, and I think that we can probably (emphasis) move forward with writing an RFC and getting it implemented. I only speak for myself though.

[CBenoit]

I’m on the same page

[gavynriebau]

How would a plugin system built around WASM would work from the perspective of a user of helix who wants to install and use plugins?

WASM is a binary format so my understanding is that each plugin would need to be compiled before it could be used. Because of this it wouldn't be possible to simply add a list of git repo URLs to a configuration file to install plugins in the same manner as for example packer.nvim.

Would plugin authors compile their plugins so that consumers don't have to? If so where would the compiled binaries be stored / would a central plugin repository (sort of like npmjs.org) need to be created?

[kirawi]

We should also be able to host something on GitHub like other open-source package managers. Though I suppose a concern would be that malicious code could end up in there.

[Th3Whit3Wolf]

@kirawi what are your thoughts on a self hosted registry? I know Helix has enough todo as it is but it would be super cool if Helix had an open source registry like vscode where users can search for plugins. I think there's enough interest this project that people would be willing to fund it on open collective.

[kirawi]

I'm not opposed to it, but I don't have the authority to say if it'll happen either. Server costs and all. That's why I was thinking of hosting it as a GitHub repo like nix or scoop

[Nehliin]

I think hosting plugins on github is a good idea. Malicious code is always an issue no matter where its hosted unless we expect to review every single plugin update which obviously doesn't scale.

However I would like to also bring up versioning of plugins. If the plugins are hosted as crates on crates.io for example we would get that for free and it would fit quite neatly into the rust ecosystem. I know that we want to support more languages than rust in the future but even in that case I'd strongly suggest that we enforce some kind of versioning system for plugin updates. One of my biggest pain points of vim-plug for example is that any PlugUpdate might contain breaking changes. Using github releases could be one way of solving that.

[gavynriebau]

I just wanted to point out that there are existing registries for webassembly packages that might be suitable, e.g.
https://wapm.io/

One simple idea for how this could potentially work is users could install packages using a command such as cd ~/.config/helix/plugins && wapm install hx-plugin-tree-file-explorer.

hx could be extended to start all plugins in this directory when it starts.

[7ombie]

Is this still happening? I don't know Rust, but know WebAssembly pretty well (I wrote an assembler for it). I could help with the design at least.

[the-mikedavis]

No, see https://github.com/helix-editor/helix/wiki/FAQ#how-to-write-plugins-is-there-a-plugin-system-in-place-yet

[kirawi]

PR is #8675

[7ombie]

Steel?? That'll be divisive.