Optimize bitmaps for more memory-efficiency #267
Conversation
|
I see your point that the effect seems small for the test data set if one looks only at storage consumption. There the static overhead mostly hides the effect because of the few sequences. Although, looking at the |
…st numerous symbol is deleted
Taepper
force-pushed
the
optimize-bitmaps
branch
from
851e241
to
1c8e3b9
Compare
| const auto deleted_symbol = current_position.getDeletedSymbol(); | ||
| if (deleted_symbol.has_value() && symbol != *deleted_symbol) { |
There was a problem hiding this comment.
Is this different from !current_position.isSymbolDeleted(symbol)?
| bitmap.runOptimize(); | ||
| bitmap.shrinkToFit(); |
There was a problem hiding this comment.
I don't know whether it's good to hide this side effect in a getSomething method.
| for (const uint32_t idx : *filter) { | ||
| const roaring::Roaring& n_bitmap = sequence_store_partition.missing_symbol_bitmaps[idx]; | ||
| if (n_bitmap.contains(position)) { | ||
| count_of_mutations_per_position[*deleted_symbol][position] -= 1; | ||
| } | ||
| } |
There was a problem hiding this comment.
As discussed, let's inline this into addPositionToMutationCountsForMixedBitmaps and get rid of the separate correct... method.
…here Position invariant was broken because of 'missing' symbols
Taepper
force-pushed
the
optimize-bitmaps
branch
from
048f5d1
to
c2a8439
Compare
The very memory efficient mode is now implemented:
The most numerous bitmap is deleted instead of flipped. When looking up values, all other bitmaps are considered instead.
A good next step would be a hybrid mode, where the deletion is only preferred over flipping, if there is substantial memory gains, as the performance impact is quite high