[move][aptos-vm] End-to-end module publishing

[georgemitenkov]

Description

This PR introduces module publishing for loader V1 and code cache. It also refactors some code introduces by previous PR, in order to minimize overall changes (w.r.t. to main). In particular:

1. Brought back publishing APIs used by loader V1. They are just marked as #[deprecated], but otherwise there was no reason to remove them :) It is easier to instead add a switch based on V1/V2 case rather than change both.
2. Refactored vm_harness back to minimize changes... Previous code was too different from main for no reason. Note that the actual diff from main is not visible here.
3. Introduced module publishing flow for loader v2. In short, we publish modules directly to temporary storage, which stages them and verifies if they are correct. I think it is done rather nicely, avoiding any extra traversal implementations:
   • Modules are placed as bytes in a state view like storage, under existing unsync module storage.
   • As they are put inside, we perform compat checks.
   • We then create a new unsync module storage wrapping updated byte-level view, and try to load all verified modules. If there is a failure - publish is not possible.
4. Tests are adjusted to work with the new implementation. Note that they are partially broken. See PR [move] Fixing unit tests for loader V2 #14397 which makes many Move/Aptos tests pass (and in particular, all tests under move-vm)

Type of Change

• New feature
• Bug fix
• Breaking change
• Performance improvement
• Refactoring
• Dependency update
• Documentation update
• Tests

Which Components or Systems Does This Change Impact?

• Validator Node
• Full Node (API, Indexer, etc.)
• Move/Aptos Virtual Machine
• Aptos Framework
• Aptos CLI/SDK
• Developer Infrastructure
• Other (specify)

How Has This Been Tested?

See #14397

Key Areas to Review

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

[trunk-io]

⏱️ 4h 38m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-move-unit-coverage	22m	🟩
general-lints	19m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-move-unit-coverage	19m	🟩
rust-cargo-deny	18m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-move-unit-coverage	17m	🟩
rust-move-unit-coverage	17m	⬜
rust-move-tests	14m	🟥
check-dynamic-deps	14m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-move-tests	14m	🟥
rust-move-unit-coverage	13m	🟩
rust-move-tests	13m	🟥
rust-move-tests	13m	🟥
rust-move-tests	13m	🟥
rust-move-unit-coverage	9m	🟩
rust-move-unit-coverage	9m	⬜
rust-move-unit-coverage	9m	🟩
rust-move-tests	9m	⬜
rust-move-unit-coverage	8m	🟩
semgrep/ci	4m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-move-tests	3m	🟥
rust-move-tests	3m	🟥
rust-move-tests	3m	🟥
rust-move-tests	2m	⬜
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+6 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+6 more)
rust-move-tests	2m	⬜
rust-move-unit-coverage	2m	⬜
permission-check	40s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	39s	🟩 🟩 🟩 🟩 🟩 (+7 more)
permission-check	35s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	34s	🟩 🟩 🟩 🟩 🟩 (+6 more)

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[georgemitenkov]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [move] Fixing unit tests for loader V2 #14397 : 2 dependent PRs (#14001 , #14402 )
• [move][aptos-vm] End-to-end module publishing #14395 👈
• Introduce code cache and loader V2 #14184 : 6 other dependent PRs (#14075 , #14206 , #14341 and 3 others)
• [move] Change interpreter to use loaded func and improve scoping #14185
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @georgemitenkov and the rest of your teammates on Graphite

[codecov]

Codecov Report

Attention: Patch coverage is 40.94077% with 339 lines in your changes missing coverage. Please review.

Please upload report for BASE (george/init-module@83c8549). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...rty/move/move-vm/runtime/src/storage/publishing.rs	0.0%	160 Missing ⚠️
...torage/implementations/unreachable_code_storage.rs	0.0%	51 Missing ⚠️
aptos-move/aptos-vm/src/aptos_vm.rs	0.0%	40 Missing ⚠️
...a/transactional-test-runner/src/vm_test_harness.rs	84.6%	28 Missing ⚠️
third_party/move/move-vm/runtime/src/session.rs	74.1%	8 Missing ⚠️
...rc/module_and_script_storage/state_view_adapter.rs	0.0%	7 Missing ⚠️
third_party/move/move-vm/runtime/src/runtime.rs	81.5%	7 Missing ⚠️
...rd_party/move/move-vm/runtime/src/storage/dummy.rs	0.0%	7 Missing ⚠️
...src/storage/implementations/unsync_code_storage.rs	0.0%	7 Missing ⚠️
types/src/transaction/module.rs	0.0%	6 Missing ⚠️
... and 6 more

Additional details and impacted files

@@                  Coverage Diff                  @@
##             george/init-module   #14395   +/-   ##
=====================================================
  Coverage                      ?    59.5%           
=====================================================
  Files                         ?      856           
  Lines                         ?   208742           
  Branches                      ?        0           
=====================================================
  Hits                          ?   124386           
  Misses                        ?    84356           
  Partials                      ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[georgemitenkov]

All comments will be addresses on top of george/main branch