Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Allow the allocator to track the heap changes. #9291

Merged
5 commits merged into from
Jul 7, 2021
Merged

Conversation

kianenigma
Copy link
Contributor

This will greatly help with try-runtime's offchain_worker. I can now re-execute the last offchain worker's election code, as such, and get this nice report of heap usage:

    Finished dev [unoptimized + debuginfo] target(s) in 2m 23s
     Running `target/debug/substrate try-runtime --block-at 0x32f4f40c9bf09e7406721c562356f6b3ca718fc405a0164cbca31da6fd8ece7a --execution Wasm --url 'ws://kianenigma-archive:9944' offchain-worker live -m ElectionProviderMultiPhase`
2021-07-06 20:51:23.441  INFO main remote-ext: initializing remote client to "ws://kianenigma-archive:9944"
2021-07-06 20:51:23.517  INFO main remote-ext: scraping key-pairs from remote @ 0x32f4f40c9bf09e7406721c562356f6b3ca718fc405a0164cbca31da6fd8ece7a
2021-07-06 20:51:23.580 DEBUG main remote-ext: last page received: 6
2021-07-06 20:51:23.580  INFO main remote-ext: Querying a total of 6 keys
2021-07-06 20:51:24.928  INFO main remote-ext: downloaded data for module ElectionProviderMultiPhase (count: 6 / prefix: ede8e4fdc3c8b556f0ce2f77fc2575e3).
2021-07-06 20:51:24.928 DEBUG main remote-ext: adding data for hashed key: 26aa394eea5630e07c48ae0c9558cef7f9cce9c888469bb1a0dceaa129672ef8
2021-07-06 20:51:24.968 DEBUG main remote-ext: adding data for hashed key: 3a636f6465
2021-07-06 20:51:25.404  INFO main remote-ext: extending externalities with 0 manually injected key-values
2021-07-06 20:51:25.405  INFO main remote-ext: injecting a total of 8 keys
2021-07-06 20:51:28.527 DEBUG main wasm-heap: allocator being destroyed, max total_size 800, max bumper 1269856
2021-07-06 20:52:14.502 DEBUG main wasm-heap: allocator being destroyed, max total_size 29645680, max bumper 70057920
2021-07-06 20:52:14.505  INFO main try_runtime_cli: OffchainWorkerApi_offchain_worker executed without errors.

which shows the bumper being around 70mb, still far from the current 128mb.

@kianenigma kianenigma added A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit. labels Jul 6, 2021
client/allocator/src/freeing_bump.rs Outdated Show resolved Hide resolved
client/allocator/src/freeing_bump.rs Outdated Show resolved Hide resolved
client/allocator/Cargo.toml Outdated Show resolved Hide resolved
}
macro_rules! log {
($level:tt, $( $args:expr ),+ ) => {
log::$level!(target: "wasm-heap", $( $args ),+);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH, this macro looks (at the call sites) a bit weird. Why don't we just remove it altogether, and stick to a regular logging as found in the rest of codebase?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a patter that I use a lot in pallets, but I will respect the fact that this is the client realm and most logs are not like this, so will revert.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, my main preference for this is that I don't like repeating target: "..." all over the place.

@@ -26,5 +26,5 @@ std = [
"sp-std/std",
"sp-core/std",
"sp-wasm-interface/std",
"log",
"log/std",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"log/std",

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh about this: now the allocator is even in client. Should this crate really have a separate "std" feature? I can just remove that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The std feature is not needed. We should remove it IMO

client/allocator/Cargo.toml Outdated Show resolved Hide resolved
kianenigma and others added 2 commits July 6, 2021 21:47
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Copy link
Contributor

@emostov emostov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with the caveat that I don't feel qualified to fully asses the performance implications.

@kianenigma
Copy link
Contributor Author

bot merge

@ghost
Copy link

ghost commented Jul 7, 2021

Trying merge.

@ghost ghost merged commit 933e9c5 into master Jul 7, 2021
@ghost ghost deleted the kiz-track-allocator branch July 7, 2021 05:32
This pull request was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A0-please_review Pull request needs code review. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. D2-notlive 💤 PR contains changes in a runtime directory that is not deployed to a chain that requires an audit.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants