Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tracking Issue] Flat storage for resharding V3 #12174

Open
9 of 24 tasks
Trisfald opened this issue Sep 30, 2024 · 2 comments
Open
9 of 24 tasks

[Tracking Issue] Flat storage for resharding V3 #12174

Trisfald opened this issue Sep 30, 2024 · 2 comments
Assignees

Comments

@Trisfald
Copy link
Contributor

Trisfald commented Sep 30, 2024

Part of #11881

Goal

Support resharding V3 for Flat Storage. On shard layout, Flat Storage must be able to take a shard and split it into two children.
The entire process will happen in the background with no strict time requirement. Additional processing resources consumption should be minimal because, in the meantime, the node will continue to apply blocks and chunks.

Sub-tasks

  • Implementation
    • Support for splitting a Flat storage shard
      • Handle all Flat storage key-value pairs
        • Simple keys
        • Receipts-like keys
      • Possibility to interrupt resharding
      • Background processing
      • Creating and apply deltas on children shards
      • Handle parent shard deletion
    • Add observability
      • About progress of background task
      • About progress of catchup
      • Time shard deletions operations
      • Metric for status of flat storage (?)
    • Integration with ReshardingManager
    • Ensure parent flat storage is at good height
    • Handle parent flat storage deltas
    • Better limits on background task iteration, like key size
    • Trigger memtrie rebuild
  • Tests

    • Unit tests
      • Basic flat storage resharder functionality
      • Splitting simple keys
      • More complex resharding with other types of keys
      • Catchup children
      • Parent flat state height and deltas
    • Integration tests
      • To be covered with test loop resharding test
    • Test with forknet: resharding, restart
      • To be done for resharding as a whole
@Trisfald
Copy link
Contributor Author

merged #12164 tackling the introduction of FlatStorageResharder and simple key splitting

@Trisfald
Copy link
Contributor Author

merged #12223 to handle all key types

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant