Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(resharding): implemented state-stats state-viewer command #10094

Merged
merged 8 commits into from
Nov 6, 2023
Merged

feat(resharding): implemented state-stats state-viewer command #10094

merged 8 commits into from
Nov 6, 2023

Conversation

wacban
Copy link
Contributor

@wacban wacban commented Nov 3, 2023

the state-stats commands iterates the flat state and prints some simple statistics about each shard

It's a bit imperfect as it doesn't combine together different records belonging to the same account (e.g. contract, contract data, ..), but I'll leave that for another time.

For shard 3 the results would suggest that we should reshard at account "tge-lockup.sweat". I'll do further analysis.

here is an output for mainnet as of today

023-11-03T16:11:31.671944Z  INFO state_viewer: s0.v1
2023-11-03T16:11:31.671990Z  INFO state_viewer: StateStats {
    total_size: 6.5 GB,
    total_count: 45617105,
    average_size: 142 B,
    middle_state_record: StateStatsStateRecord {
        account_id: "86c7dfa3016ab2c32ecc0baf6a6a79c4ca408309.lockup.near",
        type: "Contract",
        size: 341.2 KB,
    },
    top_state_records: [
        StateStatsStateRecord {
            account_id: "7f3d1be6c7322ca5a792900b3a42d7131251628b2bbe9e1515e7b4973ede4aee",
            type: "Contract",
            size: 2.1 MB,
        },
        StateStatsStateRecord {
            account_id: "1111111111111222222222222222www22223.laboratory.jumpfinance.near",
            type: "Contract",
            size: 2.1 MB,
        },
        StateStatsStateRecord {
            account_id: "aaxxii.laboratory.jumpfinance.near",
            type: "Contract",
            size: 2.1 MB,
        },
        StateStatsStateRecord {
            account_id: "apollo42.near",
            type: "Contract",
            size: 2.3 MB,
        },
        StateStatsStateRecord {
            account_id: "atocha.octopus-registry.near",
            type: "Contract",
            size: 2.2 MB,
        },
    ],
}


2023-11-03T16:11:57.381410Z  INFO state_viewer: s1.v1
2023-11-03T16:11:57.381445Z  INFO state_viewer: StateStats {
    total_size: 5.9 GB,
    total_count: 43698264,
    average_size: 134 B,
    middle_state_record: StateStatsStateRecord {
        account_id: "aurora",
        type: "Data",
        size: 98 B,
    },
    top_state_records: [
        StateStatsStateRecord {
            account_id: "aurora",
            type: "Data",
            size: 24.6 KB,
        },
        StateStatsStateRecord {
            account_id: "aurora",
            type: "Data",
            size: 24.6 KB,
        },
        StateStatsStateRecord {
            account_id: "aurora",
            type: "Data",
            size: 24.6 KB,
        },
        StateStatsStateRecord {
            account_id: "aurora",
            type: "Contract",
            size: 1.1 MB,
        },
        StateStatsStateRecord {
            account_id: "aurora",
            type: "Data",
            size: 128.7 KB,
        },
    ],
}


2023-11-03T16:12:19.961616Z  INFO state_viewer: s2.v1
2023-11-03T16:12:19.961656Z  INFO state_viewer: StateStats {
    total_size: 5.9 GB,
    total_count: 31472909,
    average_size: 186 B,
    middle_state_record: StateStatsStateRecord {
        account_id: "gachi.tkn.near",
        type: "Contract",
        size: 213.1 KB,
    },
    top_state_records: [
        StateStatsStateRecord {
            account_id: "coto-uat.sukerman.near",
            type: "Contract",
            size: 2.2 MB,
        },
        StateStatsStateRecord {
            account_id: "craftingv1.near",
            type: "Contract",
            size: 2.3 MB,
        },
        StateStatsStateRecord {
            account_id: "envelop_wnft721.near",
            type: "Contract",
            size: 2.3 MB,
        },
        StateStatsStateRecord {
            account_id: "envelop_wrapper.near",
            type: "Contract",
            size: 2.4 MB,
        },
        StateStatsStateRecord {
            account_id: "exchange.slowisfast.near",
            type: "Contract",
            size: 2.5 MB,
        },
    ],
}


2023-11-03T16:13:16.188511Z  INFO state_viewer: s3.v1
2023-11-03T16:13:16.188550Z  INFO state_viewer: StateStats {
    total_size: 11.3 GB,
    total_count: 89305884,
    average_size: 126 B,
    middle_state_record: StateStatsStateRecord {
        account_id: "tge-lockup.sweat",
        type: "Data",
        size: 176 B,
    },
    top_state_records: [
        StateStatsStateRecord {
            account_id: "switchboard-v2.near",
            type: "Contract",
            size: 2.7 MB,
        },
        StateStatsStateRecord {
            account_id: "v2.perp.spin-fi.near",
            type: "Contract",
            size: 2.8 MB,
        },
        StateStatsStateRecord {
            account_id: "v2_0_2.perp.spin-fi.near",
            type: "Contract",
            size: 2.8 MB,
        },
        StateStatsStateRecord {
            account_id: "nearpay-portals.near",
            type: "Contract",
            size: 3.9 MB,
        },
        StateStatsStateRecord {
            account_id: "moodev.near",
            type: "Contract",
            size: 3.8 MB,
        },
    ],
}

@wacban wacban requested a review from a team as a code owner November 3, 2023 16:30
@walnut-the-cat
Copy link
Contributor

maybe we can add which shard the stat is for as well? :o Also, kinda random question. What value do we get from top state records?

@wacban
Copy link
Contributor Author

wacban commented Nov 6, 2023

maybe we can add which shard the stat is for as well? :o

If you mean to print it to the output then it is. If you mean to add a cmdline argument to select one, I don't need it right now but it can be easily added any time. I opted not to do it upfront to keep the command usage as simple as possible.

Also, kinda random question. What value do we get from top state records?

It was more of a "I can do it" rather than "I need to do it" kinda thing :) If I had to make up some reason for it, it would be: Generally speaking it would be good to understand the structure and properties of the trie a bit better.

shreyan-gupta
shreyan-gupta approved these changes Nov 6, 2023
View reviewed changes
Copy link
Contributor

@shreyan-gupta shreyan-gupta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I just hope we aren't duplicating anything from what Robin has.

tools/state-viewer/src/commands.rs

let iter = chunk_view.iter_flat_state_entries(None, None);
for item in iter {
let Ok((key, value)) = item else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this syntax! Didn't know it existed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Looking at the implementation of iter_flat_state_entries, it might be fine to just unwrap key value here but can leave as is.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sytax is cool yeah, just learned it recently too. :)
Actually I wanted to print a warning in there, let me do that.

@shreyan-gupta
Copy link
Contributor

shreyan-gupta commented Nov 6, 2023 •
edited
Loading

Also not 100% sure what exactly would be the use of top_state_records? It's supposed to be bound by the contract size right?

@wacban wacban enabled auto-merge November 6, 2023 10:32
@wacban
Copy link
Contributor Author

wacban commented Nov 6, 2023

@shreyan-gupta What do you mean here?

It's supposed to be bound by the contract size right?

shreyan-gupta
shreyan-gupta reviewed Nov 6, 2023
View reviewed changes
tools/state-viewer/src/commands.rs
}
}

fn read_flat_state_value(
Copy link
Contributor

@shreyan-gupta shreyan-gupta Nov 6, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the value itself is used anywhere... Quick optimization, we can perhaps just work with the size of the value instead of the vector value itself? But can leave as is

@wacban wacban added this pull request to the merge queue Nov 6, 2023
Merged via the queue into master with commit 2dd724e Nov 6, 2023
21 of 22 checks passed
@wacban wacban deleted the waclaw-resharding-boundary branch November 6, 2023 11:09
@wacban wacban mentioned this pull request Nov 16, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shreyan-gupta shreyan-gupta shreyan-gupta approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wacban @walnut-the-cat @shreyan-gupta