Skip to content
This repository has been archived by the owner on Jun 6, 2023. It is now read-only.

Implement load balancing of partitions among a miner's deadlines #409

Closed
anorth opened this issue Jun 3, 2020 · 2 comments · Fixed by #438
Closed

Implement load balancing of partitions among a miner's deadlines #409

anorth opened this issue Jun 3, 2020 · 2 comments · Fixed by #438
Assignees
Labels
code-ready Ready for implementation, with clear path P1 High priority, required for basic network functionality and growth scale Related to chain state or throughput scalability
Milestone

Comments

@anorth
Copy link
Member

anorth commented Jun 3, 2020

Follow-on work from the initial Window PoSt implementation is to assign new sectors to deadlines so as to balance the load for a single miner between its deadlines.

A simple sort (as suggested in #402) is attractive, but not correct if there are a larger number of partitions to be filled than deadlines of equally-minimal weight. The full solution probably involves building a heap or maintaining a sorted list in order to repeatedly add the next partition to a minimal deadline, taking into account partitions just added.

See also #402 and #432.

@anorth anorth added P1 High priority, required for basic network functionality and growth changes state scale Related to chain state or throughput scalability labels Jun 3, 2020
@anorth
Copy link
Member Author

anorth commented Jun 4, 2020

I've broken the randomisation out to #432, as it is lower priority than simply load-balancing. I think that means that the fix for #402 will resolve this too. cc @ZenGround0

@anorth anorth added the code-ready Ready for implementation, with clear path label Jun 4, 2020
@anorth
Copy link
Member Author

anorth commented Jun 4, 2020

Quick estimates: for a miner to onboard 1EiB of storage in 1 year with 32GiB sectors, they'll be filling about 40 partitions per proving period. This puts it in the range where deciding between the O(n^2) maintenance of a sorted list vs O(n log n) but more complicated maintenance of a heap isn't clear. The heap is overkill for miners onboarding at half that rate.

64GiB sectors will halve that, so maybe go with the simple sorted list (example).

@ZenGround0 ZenGround0 self-assigned this Jun 4, 2020
@anorth anorth self-assigned this Jun 5, 2020
@anorth anorth added this to the Release 0.6 milestone Jun 15, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
code-ready Ready for implementation, with clear path P1 High priority, required for basic network functionality and growth scale Related to chain state or throughput scalability
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants