Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: variable vnode count support in table distribution #18373

Merged
merged 4 commits into from
Sep 5, 2024

Conversation

BugenZhao
Copy link
Member

@BugenZhao BugenZhao commented Sep 3, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This is a progress towards #15900.

Specifically in the upcoming PRs, we aim to decrease the direct references on VirtualNode::COUNT within the codebase. The total vnode count will be provided as a parameter in the refactored functions, although it will remain hardcoded as VirtualNode::COUNT at the call-site. Eventually, it will be made user-facing.

This PR refactors the TableDistribution. Logic changes are:

  • Will use bitmaps with all high bits and of length 1 for singleton table. Note that it's not 256 any more.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@graphite-app graphite-app bot requested a review from a team September 3, 2024 08:09
@BugenZhao BugenZhao changed the title WIP: feat: variable vnode count in table distribution feat: variable vnode count in table distribution Sep 3, 2024
@BugenZhao BugenZhao changed the title feat: variable vnode count in table distribution feat: variable vnode count support in table distribution Sep 3, 2024
use crate::hash::VirtualNode;
use crate::row::Row;
use crate::util::iter_util::ZipEqFast;

/// For tables without distribution (singleton), the `DEFAULT_VNODE` is encoded.
pub const DEFAULT_VNODE: VirtualNode = VirtualNode::ZERO;
pub use DEFAULT_VNODE as SINGLETON_VNODE;

use super::VnodeBitmapExt;

#[derive(Debug, Clone)]
enum ComputeVnode {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May just rename it to TableDistribution, because the TableDistribution only wraps compute_vnode now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure whether we could introduce more common fields in the future. Up to @wenym1 to decide.

src/common/src/hash/table_distribution.rs Outdated Show resolved Hide resolved
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@BugenZhao BugenZhao added this pull request to the merge queue Sep 5, 2024
Merged via the queue into main with commit 670a94f Sep 5, 2024
30 of 31 checks passed
@BugenZhao BugenZhao deleted the bz/var-vnode-table-dist branch September 5, 2024 06:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants