taxonomy-from-table memory usage

[colinbrislawn]

I'm testing out the new GreenGenes! 💚

The taxonomy-from-table is currently being killed once it reaches about 50% memory usage on a 16 GB VM.

The introduction post warned about this

NOTE: Just like filter-features, this command right now will require around 8-10GB of memory.

It's possible my system is overly aggressive with memory management, but either way I'm interested in tracking this issue for all the folks with potatoes

[wasade]

Thanks, @colinbrislawn!

Ya at the moment the method is quite burdensome, even with the short cuts we already implemented. Its original implementation was much worse in memory... What I'm considering is representing the taxonomic data in a SQLite3 database which would avoid the resident overhead, and likely would not greatly impacting performance. My hope is to have this in for the next release which I'm currently working on the upstream pieces for.

Out of curiosity, is this something you'd have time and interest in working on?

[colinbrislawn]

Maybe. I've not worked with SQLite3 before so I'm not sure I'm the best fit.

I can help with testing or docs, but that's mostly done for this plugin.

[wasade]

Okay, no worries, totally understand!