support having thousands of tracks

[cmdcolin]

Having thousands of tracks introduces some issues of scale. We may need react-window to reduce number of dom elements in track selector, or we may see mobx-state-tree performance issues, or other unexpected things

Create something like modencode test dataset, or larger

[cmdcolin]

over 2500 tracks here https://raw.githubusercontent.com/KiddLab/kmer_1KG/master/kmer-1KG.hub.txt

mentioned in gitter recently, source
from https://twitter.com/JeffreyMKidd/status/1222532626520522753

[cmdcolin]

Here is another hub, lots of tracks

https://resources.altius.org/~jvierstra/projects/footprinting.2020/

One option could be a virtualized tree (another could be using more simple input types but i think even then dom node count could dominate over time)

I made a short demo here using react-virtualized-tree

Code https://github.com/GMOD/jbrowse-components/tree/virtualized_tree

Handles 1700 tracks from this connection pretty easily, and even the one above, totaling 4000 tracks

[cmdcolin]

here is a link to look and feel of the virtualized tree http://s3.amazonaws.com/jbrowse.org/code/jb2/alpha/virtualized_tree/index.html?config=test_data%2Fvolvox%2Fconfig.json

pretty raw state, just a preview though

[cmdcolin]

as mentioned, has much better scalability though to many tracks

[cmdcolin]

could possibly render full material ui components in the virtualized tree too, to match the current look and feel

[cmdcolin]

copy of instance from @shihabdider
http://hg38.jbrowse.org/jb2

exhibits slowness

probably gonna go ahead with the virtualized selector in some shape or form to achieve speedup

[shihabdider]

I started using the new test track selector with good results (no more slowdown)
http://hg38.jbrowse.org/jb2_track_test

Unfortunately, the filter tracks text box seems not be integrated yet. Typing in a search query does nothing. Also, the word wrap on the longer track names seems to be broken so that track names will overlap.

[cmdcolin]

If you have any interest in working on this issue it would be pretty exciting...it is unclear at this point if this virtualized track selector is "good enough" or "stylish enough" to replace our default track selector..it might not happen before our v1 release and then at that point it can be hard to make dramatic shifts in our styling

[cmdcolin]

The other alternative is the virtualized track selector is maybe a non default one for specialized instances

[cmdcolin]

Refer to https://github.com/GMOD/jbrowse-components/pull/1243/files for a previous attempt

[cmdcolin]

Problems with that PR included

1. virtualized autosizer did not work to expand the virtual scroll area to the region available on the side bar. This went over my head but might be related to CSS. This aspect is important to make the UI feel right because the virtual scroll area has to expand to take up some area, but avoid having nested scroll areas in the sidebar. The PR basically just guessed at the proper virtualized scroll area using window.height - fudgeFactor basically
2. Long track labels would not wrap or create a multi-line entry, which would be generally difficult with virtualized stuff as then rows take variable height
3. then extra stuff which was noted like filters not working

[cmdcolin]

Also note

1. Did not even try to NOT use virtualized and instead remove complicated MUI styling
2. https://github.com/bvaughn/react-virtualized/blob/master/docs/AutoSizer.md

[cmdcolin]

We will probably keep pushing on it with the tree. I'll create an updated post with current status

[cmdcolin]

I made a demo with 1000genomes data, with their new "expanded set" of trio data, because i thought it was a great interesting dataset to load...paper https://www.biorxiv.org/content/10.1101/2021.02.06.430068v1

It is really a pretty interesting demo because it has SV genotypes for 500+ family trios (mom, dad, child), which is a great dataset to explore (you can see whether the child inherited the structural variant or not)

Example
http://s3.amazonaws.com/jbrowse.org/code/jb2/master/index.html?config=https%3A%2F%2Fs3.amazonaws.com%2Fjbrowse.org%2Fgenomes%2FGRCh38%2F1000genomes%2Fconfig_1000genomes.json&session=share-eDTARlUJ3Q&password=Elq23

the tracklist is currently a big pain point, hence noting here....

[cmdcolin]

one small breakthrough on this branch: to properly use the react-virtualized-auto-sizer (which correctly figures out the proportions to maximize the area to put the virtualized scroller,helps to fill the area in the drawer), we don't want to have an empty div wrapping the tree

this is kind of silly but this fails:

<DrawerWidget>
<div>
<Autosizer>
<VirtualizedTree>
</Autosizer>
</div>
</DrawerWidget>

while this works

<DrawerWidget>
<Autosizer>
<VirtualizedTree>
</Autosizer>
</DrawerWidget>

not really sure why. This was also tricky to track down because the empty div was due to a non-existant className being used e.g.

still a little bit weird autosize behavior, have to subtract the height of the toolbar from what the autosizer gives, but this is closer than what we had before where the autosizer did not work

this now works nice and can quickly scroll large datasets

same 1000 genomes example on this branch

http://s3.amazonaws.com/jbrowse.org/code/jb2/virtualized_tree/index.html?config=https%3A%2F%2Fs3.amazonaws.com%2Fjbrowse.org%2Fgenomes%2FGRCh38%2F1000genomes%2Fconfig_1000genomes.json&session=share-eDTARlUJ3Q&password=Elq23

[cmdcolin]

random demo: interesting workflow where you inspect the trio for SVs following steps like this

a) go to a variant of interest on VCF track
b) click on feature details
c) load up the 3000 rows in this table
d) sort table by genotype
e) find a non-zero genotype of interest from sorting it
f) double click the name of the sample and copy and paste it
g) go to the tracklist
h) paste sample name in tracklist
i) quickly filters tracklist...then filter on category also to get full trio
j) observe the pattern of structural variant inheritance in your sample in the trio

Example video doing this

https://youtu.be/zy3nC4OfP2U