Make Export Less RAM Intensive (IAVL Changes)

[czarcas7ic]

Background

It currently takes VMs with 64GB of RAM to export/import full state exports. Some basic IAVL improvements can be done to improve this performance.

Suggested Design

• Key format change
  • Replace all native maps with BTree maps
• We currently have atmoic writes at a per module scope, which is not efficient
  • We should lower the scope of atomic writes from the 100s of MBs to the 100s of KBs

Acceptance Criteria

• Import/Export does not take extraordinary amounts of RAM to work

[catShaark]

Can I have some questions ?

• What native maps should we replace ? Maybe maps like this ?
• We should use https://github.com/tidwall/btree for BTree maps right ?

[catShaark]

We currently have atomic writes at a per module scope

This refers to cms Store's Write() right ? Can you explain this more ?

[czarcas7ic]

I am going to have to have a separate conversation with Dev to get a better / more concrete understanding of the best way forward with this. This might have to happen later this week sadly

[catShaark]

seems like the non-atomic writes is explained here

[catShaark]

@czarcas7ic, do we have any update on this?

[czarcas7ic]

I sadly do not have context on this, @ValarDragon would you be able to write something out in order to allow @catShaark to pick up this task?

[ValarDragon]

Key Format Changes

Basically make key formats we write to disk follow write_version | path_to_node

Replace all native maps with BTree maps

This is entirely unrelated

We currently have atmoic writes at a per module scope, which is not efficient

Basically just change the batch.set & batch.writeSync usage in IAVL, to write in many smaller batches. E.g. write every 1MB atomically, don't keep growing the batches. We already have tooling for detecting when we've completed the final write, the whole thing should not be atomic, it should just be possible to safely automatically rollback / wipe latest version.

[p0mvn]

@catShaark I would love to work with you on smaller batch writing. Would you be interested in working together?

[catShaark]

yesss, just tell me what to do

[tac0turtle]

why not streaming of genesis? agree that batching would help but also streaming would be the correct solution

[ValarDragon]

Unneeded due to in-place testnet