Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(resharding): state-stats improvements #10119

Merged
merged 2 commits into from
Nov 7, 2023
Merged

feat(resharding): state-stats improvements #10119

merged 2 commits into from
Nov 7, 2023

Conversation

wacban
Copy link
Contributor

@wacban wacban commented Nov 7, 2023 •
edited
Loading

  • group records by account id and aggregate it to get the total size of all records belonging to each account is used
  • re-arrange records to be sorted by account id, rather than type and account id

results on mainnet

2023-11-07T13:25:00.063905Z  INFO state_viewer: s0.v1
2023-11-07T13:25:00.063968Z  INFO state_viewer: StateStats {
    total_size: 6.5 GB,
    total_count: 18507133,
    average_size: 351 B,
    middle_state_record: StateStatsStateRecord {
        account_id: "66fde23ac459779474beb6d1f8b72164e67946de.lockup.near",
        size: 344.9 KB,
    },
    top_accounts: [
        StateStatsStateRecord {
            account_id: "athlete.promotional.basketball.playible.near",
            size: 10.0 MB,
        },
        StateStatsStateRecord {
            account_id: "amm.counselor.near",
            size: 11.7 MB,
        },
        StateStatsStateRecord {
            account_id: "asset-manager.orderly-network.near",
            size: 142.2 MB,
        },
        StateStatsStateRecord {
            account_id: "anthropocene.mintbase1.near",
            size: 18.0 MB,
        },
        StateStatsStateRecord {
            account_id: "alpha-x.raddx.near",
            size: 405.3 MB,
        },
    ],
}


2023-11-07T13:29:24.136367Z  INFO state_viewer: s1.v1
2023-11-07T13:29:24.136422Z  INFO state_viewer: StateStats {
    total_size: 5.9 GB,
    total_count: 1,
    average_size: 5.9 GB,
    middle_state_record: StateStatsStateRecord {
        account_id: "aurora",
        size: 5.9 GB,
    },
    top_accounts: [
        StateStatsStateRecord {
            account_id: "aurora",
            size: 5.9 GB,
        },
    ],
}


2023-11-07T13:32:37.941254Z  INFO state_viewer: s2.v1
2023-11-07T13:32:37.941309Z  INFO state_viewer: StateStats {
    total_size: 5.9 GB,
    total_count: 10006107,
    average_size: 587 B,
    middle_state_record: StateStatsStateRecord {
        account_id: "deadmau5.mintbase1.near",
        size: 168.2 MB,
    },
    top_accounts: [
        StateStatsStateRecord {
            account_id: "canvas-war.feiyu.near",
            size: 50.5 MB,
        },
        StateStatsStateRecord {
            account_id: "jars.sweat",
            size: 77.9 MB,
        },
        StateStatsStateRecord {
            account_id: "fusotao.octopus-registry.near",
            size: 51.8 MB,
        },
        StateStatsStateRecord {
            account_id: "deadmau5.mintbase1.near",
            size: 168.2 MB,
        },
        StateStatsStateRecord {
            account_id: "bridge-validator.sputnik-dao.near",
            size: 134.4 MB,
        },
    ],
}

2023-11-07T13:40:05.003453Z  INFO state_viewer: s3.v1
2023-11-07T13:40:05.003511Z  INFO state_viewer: StateStats {
    total_size: 11.3 GB,
    total_count: 3113322,
    average_size: 3.6 KB,
    middle_state_record: StateStatsStateRecord {
        account_id: "tge-lockup.sweat",
        size: 3.5 GB,
    },
    top_accounts: [
        StateStatsStateRecord {
            account_id: "wallet.kaiching",
            size: 158.4 MB,
        },
        StateStatsStateRecord {
            account_id: "nft.nearapps.near",
            size: 321.6 MB,
        },
        StateStatsStateRecord {
            account_id: "x.paras.near",
            size: 605.6 MB,
        },
        StateStatsStateRecord {
            account_id: "token.sweat",
            size: 1.4 GB,
        },
        StateStatsStateRecord {
            account_id: "tge-lockup.sweat",
            size: 3.5 GB,
        },
    ],
}

@wacban wacban requested a review from a team as a code owner November 7, 2023 13:17
shreyan-gupta
shreyan-gupta approved these changes Nov 7, 2023
View reviewed changes
Copy link
Contributor

@shreyan-gupta shreyan-gupta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add the results from execution as well?

tools/state-viewer/src/commands.rs Outdated
impl Iterator<Item = StateStatsStateRecord> + 'a,
impl FnMut(&StateStatsStateRecord) -> AccountId,
> {
// let iter = chunk_view.iter_flat_state_entries(None, None);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove

tools/state-viewer/src/commands.rs
}
}

// Gets the flat state iterator and groups items by account id.
fn get_state_stats_group_by<'a>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aah this is very clever! Took me some time to understand what's going on, had to read up on the kmerge and group_by functions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It took me real long to get the types, lifespans and whatnots in order but yeah it's pretty now :)

tools/state-viewer/src/commands.rs
.field("size", &self.size())
.finish()
// Finally, group by the account id.
iter.group_by(|state_record| state_record.account_id.clone())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would group_by &state_record.account_id work here?

@wacban
Copy link
Contributor Author

wacban commented Nov 7, 2023

Could you please add the results from execution as well?

Added to the PR description.

@wacban wacban added this pull request to the merge queue Nov 7, 2023
Merged via the queue into master with commit f229c33 Nov 7, 2023
16 of 17 checks passed
@wacban wacban deleted the waclaw-resharding-state-stats branch November 7, 2023 14:20
This was referenced Nov 16, 2023
Closed
Merged
github-merge-queue bot pushed a commit that referenced this pull request Nov 17, 2023
@wacban
feat(resharding): set the shard boundary in the new shard layout (#10196
1b0bfca 
)

based on the analysis done in #10119
it's not yet set in stone but it's the most likely candidate so let's
set it and test it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shreyan-gupta shreyan-gupta shreyan-gupta approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wacban @shreyan-gupta