Update developer guide for 1.0.3 plugin system

[cmdcolin]

No description provided.

[garrettjstevens]

Are there any specific sections you noticed that need to be updated? Note, some changes were made in edf5833.

[cmdcolin]

I guess I thought we might need to do more usage of jbrequire and factory functions e.g.

https://jbrowse.org/jb2/docs/developer_guide/#example-data-adapter

Might need to make factory with jbrequire from to get BaseFeatureDataAdapter since it needs precise instanceof check matching

[jjrozewicki]

I have been trying to make a plugin for JBrowse 2 for the past few weeks. I've tried a lot of different angles, but I keep running into issues with build/compilation.

I first started here:
https://jbrowse.org/jb2/docs/developer_guide/#putting-it-all-together

I noticed the exports had changed so I fixed that issue, but I couldn't get it to compile. The developer guide doesn't include any example plugin that implements a build system that can compile that example code. There are example plugins, but none of those examples are able to compile the example code I linked.

The next thing I tried was hollowing out the working UCSC plugin to see if I could get it to do what I wanted to do. I was able to remove code I didn't need while keeping it able to compile, but when I tried to implement my own version of a sequence interpreter based on the sequence plugin I again ran into build issues with trying to include the IndexedFasta @GMOD module.

The common theme is that I can almost always get the CJS and ESM versions to compile, but the second I try to do anything complicated tsdx refuses to pack the UMD version. The Developer Guide talks about the plugin build system and how plugins can be distributed as NPM modules, but the linked example plugins are all UMD side-loaded via URL in the config. ESM/CJS-formatted plugins error when loaded that way.

The basic thing I want to do is fork the existing sequence plugin to display custom statistics via LinearWiggleDisplay. I've been trying to use the existing add-on plugins, the guide, and the plugins included in the core package (wiggle, sequence, etc.) as a guide, but I have not really been able to make any progress.

I didn't post here sooner because I'm not very familiar with the JavaScript ecosystem and I couldn't puzzle out what was an issue with the build system, the API, my environment, my lack of JS knowledge, etc. So I've spent a lot of time Googling for general JavaScript issues, but none have been able to fix my problems. The thing I keep running into is that most of the info applies to workflows that use webpack and rollup directly, and not tsdx.

I would like to resolve this issue because my current project could benefit a lot from using JBrowse2 over JBrowse1, but I'm not seeing a clear path forward other than starting from an empty plugin stub and writing my plugin in raw JS to sidestep the build/packaging issues. I'd like to avoid that because it's not super sustainable long term, but that's kind of where I'm at.

[elliothershberg]

Hey @jjrozewicki thanks for reaching out, it's great to hear that JBrowse 2 seems like a good fit for your project, and that you are interested in developing a plugin. You are definitely right that there have been some build changes to the plugin system since those guides. We are working on improving the workflow right now actually.

Have you taken a look at the plugin template yet? It can be found here: https://github.com/GMOD/jbrowse-plugin-template

This template provides a starting point for plugin development that works with the new changes to the build system. We are also getting ready to add some testing functionality and tools to the template sometime this week: GMOD/jbrowse-plugin-template#2

Let us know if this helps with the compilation issues you are seeing, or if you are still running into roadblocks

[garrettjstevens]

Hi @jjrozewicki, I think I can reproduce your error. I'll look into it and keep you updated.

[garrettjstevens]

@jjrozewicki I found some things with the @gmod/indexedfasta package that weren't allowing it to be bundled. See GMOD/indexedfasta-js#47 for some more info. I also found out some more documentation we need for bundling certain packages. See below for more details.

TODO for this issue:

We need to document that if your plugin uses packages that depend on polyfills for native node modules, you may need to add some rollup plugins to your tsdx.config.js. Specifically, you'll need to add rollup-plugin-node-builtins and possibly rollup-plugin-node-globals. See here for documentation of which node builtins require you to also use rollup-plugin-node-globals: https://www.npmjs.com/package/rollup-plugin-node-builtins.

With both of these extra plugins, your tsdx.config.js would look something like this;

const globals = require('@jbrowse/core/ReExports/list').default
const { createJBrowsePluginTsdxConfig } = require('@jbrowse/development-tools')
const nodeBuiltins = require('rollup-plugin-node-builtins')
const nodeGlobals = require('rollup-plugin-node-globals')

module.exports = {
  rollup(config, options) {
    const newConfig = createJBrowsePluginTsdxConfig(config, options, globals)
    if (options.format === 'umd') {
      newConfig.plugins.push(nodeBuiltins(), nodeGlobals())
    }
    return newConfig
  },
}

[cmdcolin]

Great sleuthing @garrettjstevens

I know there are a lot of already a lot of ideas flown around here, but one more to add...I know that @jjrozewicki you mentioned the idea of plotting GC content? For a gccontent adapter, I tried to sketch out something like this here in this PR #1624

Instead of accessing @gmod/indexedfasta directly, it uses an existing jbrowse 2 dataAdapter to read the sequence data.

[garrettjstevens]

@jjrozewicki there is now a new version of the @gmod/indexedfasta package that should be able to be bundled in a plugin UMD build: https://www.npmjs.com/package/@gmod/indexedfasta

[cmdcolin]

Note that my initial concern e.g. conversion to factory functions is not strictly needed because of the jbrowse development tools packages/development-tools/index.js which allows us to import {tree} from 'mobx-state-tree' without jbrequire and it does not duplicate bundle

[cmdcolin]

Still is worth doing a refresher to the guide...and try to make it a bit less prone to bitrot

[jjrozewicki]

With the help provided here I was able to get my plugin compiling and also use the existing plugins in the project as a starting point to learn from.

I think this approach may be generally helpful to many developers, and may cut down on the need for many examples of plugins. It might be worth having a section in the documentation that's a tutorial on how to take an existing internal JBrowse2 plugin, separate it out as an external plugin, compile, and then customize.

I think this also could prevent some issues with bitrot that @cmdcolin mentioned by having less example code in documentation that needs to be updated.

[cmdcolin]

This issue was brought up in the last meeting but are there any actual todo items here? It may be that this was a confusion on my part creating this because jbrequire is not even strictly required

[cmdcolin]

There are probably doc improvements we can make...for example

• I think displays vs tracks should be covered more)
• maybe referring to external working plugins instead of the in-doc maybe-working ucsc example

[teresam856]

There are probably doc improvements we can make...for example

• I think displays vs tracks should be covered more)
• maybe referring to external working plugins instead of the in-doc maybe-working ucsc example
  @cmdcolin
  Were there any other updates (besides the ones mentioned above) to the docs we wanted to push for the 1.0.4 release or updates that we may want to have ready for 1.1 ?

[cmdcolin]

It may be that this issue is just poorly conceived. We could close it or do a review of the contents. If you want to pair on it, I'd be happy to pair review it to see if anything does need updating :)

[teresam856]

It may be that this issue is just poorly conceived. We could close it or do a review of the contents. If you want to pair on it, I'd be happy to pair review it to see if anything does need updating :)

Yeah, that would be helpful. 👍

[cmdcolin]

provisionally fixed