Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fast storage optimization for queries and iterations #5

Merged
merged 73 commits into from
Feb 11, 2022
Merged

Fast storage optimization for queries and iterations #5

merged 73 commits into from
Feb 11, 2022

Commits on Jan 12, 2022

  1. update fast node cache in set

    @p0mvn
    p0mvn committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    4dfa41e View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2022

  1. add debugging output when falling back to original logic

    @jtieri @p0mvn
    jtieri authored and p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    3d65774 View commit details
    Browse the repository at this point in the history

  2. return error instead of panic

    @jtieri @p0mvn
    jtieri authored and p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    e48528b View commit details
    Browse the repository at this point in the history

  3. WIP: refactor set ops to work with fast store

    @jtieri @p0mvn
    jtieri authored and p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    3144b6d View commit details
    Browse the repository at this point in the history

  4. update Set of mutable tree, begin unit testing

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    163f49a View commit details
    Browse the repository at this point in the history

  5. update GetVersioned to check fast nodes before trying the immutable

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    d1d7d92 View commit details
    Browse the repository at this point in the history

  6. check fast node version before nil value check in get of immutable tree

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    0e76628 View commit details
    Browse the repository at this point in the history

  7. fix small bugs and typos, continue writing unit tests for Set

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    4e53d7f View commit details
    Browse the repository at this point in the history

  8. unit test saveFastNodeVersion

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    fb33de8 View commit details
    Browse the repository at this point in the history

  9. simplify storing unsaved fast nodes

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    cdbae73 View commit details
    Browse the repository at this point in the history

  10. resolve a bug with not writing prefix for fast node to disk

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    30358a1 View commit details
    Browse the repository at this point in the history

  11. remove fast nodes from disk on save and clear fast cache when version… 

    … is deleted, fix all tests but random and with index
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    fd3a948 View commit details
    Browse the repository at this point in the history

  12. resolve an issue with randomized tests caused by the fast node cache … 

    …not being cleared when latest version is saved
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    c54050f View commit details
    Browse the repository at this point in the history

  13. split unsaved fast node changes into additions and removals

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    0195356 View commit details
    Browse the repository at this point in the history

  14. save fast node removals

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    2dcbd32 View commit details
    Browse the repository at this point in the history

  15. move fast node cache clearing to nodedb

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    f0ff650 View commit details
    Browse the repository at this point in the history

  16. use regular logic only when fast node version is greater than immutab… 

    …le tree'
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    38f0be4 View commit details
    Browse the repository at this point in the history

  17. clean up tree_random_test.go

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    196c272 View commit details
    Browse the repository at this point in the history

  18. clear unsaved fast node removals on rollback

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    2583324 View commit details
    Browse the repository at this point in the history

  19. fix randomized test failures caused by a typo in ndb DeleteVersion fo… 

    …r loop
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    6903b24 View commit details
    Browse the repository at this point in the history

  20. implement GetFast method to preserve Get with correct index return, c… 

    …onvert unit tests from Get to GetFast where applicable
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    789efe8 View commit details
    Browse the repository at this point in the history

  21. ensure Get and GetFast return the same values in tree_random_test.go

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    29775ca View commit details
    Browse the repository at this point in the history

  22. test fast node cache is live in random tree tests

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    d9d9185 View commit details
    Browse the repository at this point in the history

  23. improve mutable tree unit tests related to new functionality

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    e0afb9d View commit details
    Browse the repository at this point in the history

  24. clean up tree_test.go

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    e2661d1 View commit details
    Browse the repository at this point in the history

  25. implement GetVersionedFast to preserve the index in GetVersioned

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    bf6b2ad View commit details
    Browse the repository at this point in the history

  26. restore accidentally deleted test in mutable tree test

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    f7c6b12 View commit details
    Browse the repository at this point in the history

  27. spurious whitespace

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    b6d937f View commit details
    Browse the repository at this point in the history

  28. refactor mutable tree

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    e1915a9 View commit details
    Browse the repository at this point in the history

  29. fix comment in mutable tree

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    d2a6187 View commit details
    Browse the repository at this point in the history

  30. add benchmark results

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    ece487f View commit details
    Browse the repository at this point in the history

  31. avoid redundant full tree search in GetFast of immutable tree when fa… 

    …st node is nil and tree is latest
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    1fe01bc View commit details
    Browse the repository at this point in the history

  32. fix naming for bench test get on random keys

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    f149348 View commit details
    Browse the repository at this point in the history

  33. use method for get latestversio in get fast

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    2088ac8 View commit details
    Browse the repository at this point in the history

  34. optimize GetFast, perform a refactor to ensure that fast nodes on dis… 

    …k are matched and better test
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    5bed4af View commit details
    Browse the repository at this point in the history

  35. add latest bench

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    544e4ab View commit details
    Browse the repository at this point in the history

  36. Fast Node Iteration (#7) 

    * propagate errors from nodedb and avoid panicking

* begin implementing fast node iteration

* resolve rebase issue in random tests

* fix iteration to deserialize fast node for extracting the correct value

* finalzie iteration and let all unit tests pass

* add benchmarks

* merge GetVersioned and GetVersionedFast

* remove fast node deletion from DeleteVersion and DeleteVersionRange and benchmark

* fix and unit test iteration on mutable and immutable trees

* implement tests for iterator and iterate, begin testing fast iterator

* fix and unit test fast iterator
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    aa9f2b7 View commit details
    Browse the repository at this point in the history

  37. refactor iterate methods of mutable and immutable trees

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    727b949 View commit details
    Browse the repository at this point in the history

  38. resolve some warnings

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    84729ce View commit details
    Browse the repository at this point in the history

  39. remove old bench results

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    4997938 View commit details
    Browse the repository at this point in the history

  40. refactor bench tests for iteration

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    b4165d1 View commit details
    Browse the repository at this point in the history

  41. Fast Cache Migration (#9) 

    * implement nodedb changes to set and get chain version from the database

* implement and unit test upgrade to fast storage in mutable tree

* refactor for auto upgrade to fast version in mutable tree contructor, load version and lazy load version

* use proper functionality for getting latest version

* remove unused error

* fix comment

* use fast version value in tests

* spurious tab

* fix style problems and remove redundant code in tests
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    20a7fa6 View commit details
    Browse the repository at this point in the history

  42. Rename Get to GetWithIndex and GetFast to Get

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    d28ccac View commit details
    Browse the repository at this point in the history

  43. refactor and clean up bench tests

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    c2a89d4 View commit details
    Browse the repository at this point in the history

  44. remove extra byte from fast node length

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    682cb71 View commit details
    Browse the repository at this point in the history

  45. clean up immutable tree

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    a855368 View commit details
    Browse the repository at this point in the history

  46. refactor nil tree or ndb error for the iterators and their tests

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    f025374 View commit details
    Browse the repository at this point in the history

  47. avoid exporting methods for getting unsaved additions and removals

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    368a454 View commit details
    Browse the repository at this point in the history

  48. refactor fast upgrade to read from tree instead of traversing disk no… 

    …des and orphans, update unit tests
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    6d24a60 View commit details
    Browse the repository at this point in the history

  49. remove unneeded comment

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    5ea4c4f View commit details
    Browse the repository at this point in the history

  50. refer to storage version consistently across the project

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    92dbfd1 View commit details
    Browse the repository at this point in the history

  51. fix more warnings

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    bd1c445 View commit details
    Browse the repository at this point in the history

  52. optimize removal of fast nodes from cache on deletion

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    a424853 View commit details
    Browse the repository at this point in the history

  53. small changes in teh mutable tree

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    fa0c081 View commit details
    Browse the repository at this point in the history

  54. correct storage version key

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    d0cdc1c View commit details
    Browse the repository at this point in the history

  55. auto set fast version in SaveVersion

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    4944a59 View commit details
    Browse the repository at this point in the history

  56. avoid exporting new methods of the immutable tree

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    1319329 View commit details
    Browse the repository at this point in the history

  57. go fmt

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    40b97ee View commit details
    Browse the repository at this point in the history

  58. Fix comment in fast iterator 

    Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
    @p0mvn @ValarDragon
    p0mvn and ValarDragon committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    e9c85e0 View commit details
    Browse the repository at this point in the history

  59. add extra comment for domain in fast iterator 

    Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
    @p0mvn @ValarDragon
    p0mvn and ValarDragon committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    c85a17a View commit details
    Browse the repository at this point in the history

  60. add comments for moving the iterator before the first element

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    495ed3b View commit details
    Browse the repository at this point in the history

  61. add comment for describing what mirror is in assertIterator of testutils

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    20abc09 View commit details
    Browse the repository at this point in the history

  62. fix checking the mirror for descending iterator in tests

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    4584387 View commit details
    Browse the repository at this point in the history

  63. Update testutils_test.go with a comment 

    Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
    @p0mvn @ValarDragon
    p0mvn and ValarDragon committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    0edf216 View commit details
    Browse the repository at this point in the history

  64. Update benchmarks/bench_test.go with a comment for runKnownQueriesFast 

    Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
    @p0mvn @ValarDragon
    p0mvn and ValarDragon committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    0c1ad3f View commit details
    Browse the repository at this point in the history

  65. Update benchmarks/bench_test.go with a comment for runQueriesFast 

    Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
    @p0mvn @ValarDragon
    p0mvn and ValarDragon committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    054cdbf View commit details
    Browse the repository at this point in the history

  66. export IsFastCacheEnabled and add an assert in bench tests

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    0bd59dc View commit details
    Browse the repository at this point in the history

  67. Update comment immutable_tree.go 

    Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
    @p0mvn @ValarDragon
    p0mvn and ValarDragon committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    2312d57 View commit details
    Browse the repository at this point in the history

  68. Update comment for migration in mutable_tree.go 

    Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
    @p0mvn @ValarDragon
    p0mvn and ValarDragon committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    4f6953a View commit details
    Browse the repository at this point in the history

  69. simlify Iterate in mutable tree, add debug log for

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    e309f7f View commit details
    Browse the repository at this point in the history

  70. Fast Cache - Downgrade - reupgrade protection and other improvements (#… 

    …12)

* add leaf hash to fast node and unit test

* refactor get with index and get by index, fix migration in load version and lazy load version

* use Get in GetVersioned of mutable tree

* refactor non membership proof to use fast storage if available

* bench non-membership proof

* fix bench tests to work with the new changes

* add downgrade-reupgrade protection and unit test

* remove leaf hash from fast node

* resolve multithreading bug related to iterators not being closed

* clean up

* use correct tree in bench tests

* add cache to tree used to bench non membership proofs

* add benc tests for GetWithIndex and GetByIndex

* revert GetWithIndex and GetByIndex

* remove unused import

* unit test re-upgrade protection and fix small issues

* remove redundant setStorageVersion method

* fix bug with appending to live stage version to storage version and nit test

* add comment for setFastStorageVersionToBatch

* refactor and improve unit tests for reupgrade protection

* rename ndb's isFastStorageEnabled to hasUpgradedToFastStorage and add comments

* comment out new implementation for GetNonMembershipProof

* update comments in nodedb to reflect the difference between hasUpgradedToFastStorage and shouldForceFastStorageUpdate

* refactor nodedb tests

* downgrade tendermint to 0.34.14 - osmosis's latest cosmos sdk does not support 0.35.0

* fix bug where fast storage was not enabled when version 0 was attempted to be loaded

* implement unsaved fast iterator to be used in mutable tree (#16)
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    2dbde70 View commit details
    Browse the repository at this point in the history

  71. address comments from unsaved fast iterator PR

    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    c83f0c5 View commit details
    Browse the repository at this point in the history

  72. expose isUpgradeable method on mutable tree and unit test (#17) 

    * expose isUpgradeable method on mutable tree and unit test

* go fmt
    @p0mvn
    p0mvn committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    f0f815e View commit details
    Browse the repository at this point in the history