MRG: fix gather memory usage issue by not accumulating GatherResult

[ctb]

This is kind of a patch-fix for #2950 for sourmash gather specifically.

This PR changes sourmash gather and sourmash multigather so that they no longer store any GatherResult objects, thus decreasing memory usage substantially.

The solution is hacky at several levels, including storing a CSV file in memory rather than writing it progressively. But I think it's an important fix to get in, since gather is one of our main use cases and it's causing people some problems (including me) :(.

The PR also changes --save-matches so that it writes out sketches as they are encountered. This breaks semantic versioning a little bit because the target file for --save-matches is opened before any matches are found, and thus may be empty and may also overwrite files unnecessarily.

Ultimately, a better fix is needed - probably one that changes up the dataclasses so that they don't store MinHashes - but such a fix is beyond me at the moment.

benchmarking

with latest @ e2c199f: 645 MB

        Command being timed: "sourmash gather /home/ctbrown/transfer/SRR606249.trim.k31.sig.gz /home/ctbrown/transfer/podar-ref.zip -o xxx.csv"
        User time (seconds): 48.51
        System time (seconds): 1.15
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:49.91
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 644900
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 156
        Minor (reclaiming a frame) page faults: 254494
        Voluntary context switches: 2412
        Involuntary context switches: 2749
        Swaps: 0
        File system inputs: 31488
        File system outputs: 64
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

with this branch: 215 MB

        Command being timed: "sourmash gather /home/ctbrown/transfer/SRR606249.trim.k31.sig.gz /home/ctbrown/transfer/podar-ref.zip -o xxx.csv"
        User time (seconds): 43.38
        System time (seconds): 0.89
        Percent of CPU this job got: 97%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:45.58
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 215560
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 773
        Minor (reclaiming a frame) page faults: 148722
        Voluntary context switches: 3884
        Involuntary context switches: 6174
        Swaps: 0
        File system inputs: 151648
        File system outputs: 160
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (e2c199f) 86.61% compared to head (84f5807) 86.62%.

Files	Patch %	Lines
src/sourmash/commands.py	98.64%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           latest    #2962      +/-   ##
==========================================
+ Coverage   86.61%   86.62%   +0.01%     
==========================================
  Files         135      135              
  Lines       15262    15278      +16     
  Branches     2622     2618       -4     
==========================================
+ Hits        13219    13235      +16     
  Misses       1743     1743              
  Partials      300      300              

Flag	Coverage Δ
hypothesis-py	25.80% <1.35%> (-0.03%)	⬇️
python	92.83% <98.64%> (+<0.01%)	⬆️
rust	59.22% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ctb]

@bluegenes could you take a quick look at this and give me a smell test? If you think it's ok to merge, then please do approve it. If not, any specific feedback you have would be welcome ;)

[ctb]

One minor improvement to this PR would be to just store the CSV rows rather than the entire CSV. Feels cleaner to me.

[ctb]

ok, updated multigather too. It's no longer terribly ugly, promise!

I think this should be ready for review @sourmash-bio/devs

[bluegenes]

i think this looks good!

[ctb]

side note: this increased memory usage was present from sourmash v4.4.0 through v4.8.5; fixed in v4.8.6.