compact: Redesign compaction planning process for cost efficiency and determinism.

[bwplotka]

Related discussion: https://matrix.to/#/!WaUKIfoqfiyWQhenET:matrix.org/$16044993991172247rFDCZ:matrix.org?via=matrix.org&via=hoffie.info

Problems of the Current Algorithm

1. The current algorithm is not very efficient for the cost (e.g network bandwidth). For example, if you have blocks laid out as 2h 2h 2h 2h 2h 2h 2h 2h 2h 2h 2h 2h 2h (301 of those in total), our current planning with comp range e.g [2h, 8h, 2d, 25d] will pick the oldest 4x2h to compact 8h and so on, then same for 2d only to at then do a single block of 25d. This means 88 compactions instead of just SINGLE compaction 300x2h blocks into single 25d. Vertical compactions are totally excluded as separate compactions so it's even worse.

2. It's currently (without locking mechanism) hard to run multiple compactors on a single stream. It would be nice to apply some determinism to allow such compactor sharding across a single stream. This is however lower priority goal.

This is related to #3390

[bwplotka]

Potentially blocker as bigger compactions would require more resources: #3406

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[yeya24]

Still valid

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

Very valid

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[yeya24]

Still valid

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[vjabrayilov]

Hey, dear @yeya24 and @bwplotka. The proposal that I attached to #3406, tries to solve this issue as well. https://drive.google.com/file/d/1oZd3ENSZ7v2hONNf4pDL3yXr_0eHcwsV/view?usp=sharing

Looking forward your feedback.
Best.

[bwplotka]

Sweet, thanks, adding to my Todo list.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[Vanshikav123]

Hello ! @bwplotka @yeya24 is this issue still relevant?

[calestyo]

Sure it is, I stumbled just few days ago over the same problem, which is why I had reported #7198.

PS: Since #7198 is closed now as a duplicate of this issue here, people my also want to read the symptoms/etc. I've had reported there,... should anyone ever work on this issue.