Fold bank serialisation into serde snapshot

[svenski123]

Problem

Bank serialisation logic is currently in ledger/src/snapshot_utils and is not versioned

Summary of Changes

• Remove serde traits from Bank
• Move Bank related serde logic from snapshot_utils to serde_snapshot
• Add bank fields types representing format independent Bank persistent data
• Add crate visible ctor and field extract Bank for use by serde_snapshot

Notes

• This PR is dependent upon Cleanup error type mapping in serde_snapshot #10580
• Legacy and future snapshot formats are currently unchanged
• Future PR will remove bank fields not needed for serialisation from new future format (i.e. because fields are unused, fields can be recovered from genesis config, etc.)

[ryoqun]

@svenski123 As always, super thanks for working on this! It seems that you're really interested in this area. In that case, please be aware of my consistently-postponed attempt of establishing trust chain of snapshot #8185 I think this pr should shed lights on the ultimate milestone of fully-verifiable snapshot. :)

Also, there is random notes for snapshot security in this messy meta-tracking issue, too #7167 if interested.

[svenski123]

Rebased and dependency on #10575 removed

[svenski123]

It's not clear to me why the buildkite buid #26406 job failed - solana_cli_config::Config::import_address_labels is clearly defined in cli-config/src/config.rs and this all builds fine locally.

I will rebase to master and force push again.

[codecov]

Codecov Report

Merging #10581 into master will decrease coverage by 0.0%.
The diff coverage is 75.4%.

@@            Coverage Diff            @@
##           master   #10581     +/-   ##
=========================================
- Coverage    82.1%    82.1%   -0.1%     
=========================================
  Files         318      319      +1     
  Lines       72747    73113    +366     
=========================================
+ Hits        59777    60033    +256     
- Misses      12970    13080    +110     

[ryoqun]

I started to review this in more detail.

[ryoqun]

@svenski123 Like before, I've made some commits on top of your PR's topic branch: https://github.com/ryoqun/solana/commits/fold-bank-serialisation-into-serde-snapshot-01

Could you take a look at it? First of all, as always, I sensed that your pr is well thought-out. What I've done there wes rather cosmetic and to adjust our further direction a bit.

Before diving into the explanation of my branch, our pr's goal is...:

Make versioning Bank's (de-)serialization VERY simple even for other engineers not familiar with rather complex snapshot serialization mechanism. Make it possible for engineers to modify the Future (de-)serialization code as they wish until branching next version.

With that in mind,

My branch contains 2 commits:

• ryoqun@4a3b515: various renaming to clarify separated serialization and deserialization and to reflect structural symmetry for the two processings. Also, intentional code duplication to realize the mentioned goal. Intended to be chrry-picked by you to this pr. Also, I've added some explanations for structs.
• ryoqun@cd83759: Not intended to be cherry-picked by you. Illustration-only. This is real demand for minor Bank serialization changes. As you guessed, unused_accounts are really unused now and operating_mode was added after our mainnet-beta launch and should generally be available in banks for convenience. Note that required changes are so small and easy, thanks to your work to this day! :)

[svenski123]

Rebased to master & applied your requested patch, also worked in the abi stuff which is now in master.
AbiExample does not work with reference fields hence the workarounds.

I've aded assertions for bank fields which come from the genesis config, the intention is that these fields are removed from the snapshot entirely and the genesis config used instead.

[ryoqun]

@svenski123 Cool, CI has passed! I'll review this in depth today or tomorrow. Also, now it seems there's a merge conflict...

[ryoqun]

@svenski123 It looks like CI currently fails to compile on rustc nightly.

[svenski123]

bincode upgrade in 841ecfd broke it; now fixed.
squashed the branch and applied the renamings including the code replication.

FYI the serdes bank fields were intentionally shared between legacy and future as both those formats are "live".
Rather future was to be renamed current and then a new future snapshot format added with its own (rather smaller) serdes bank fields def in a separate PR.

Of course that was three weeks ago.

[svenski123]

Ugh the renames broke the ABI hashes...

This is beginning to feel more like a hardware design process rather than a software design process...

[ryoqun]

@svenski123 LGTM!! Thanks for your great work!!

[ryoqun]

@svenski123 I know the tedious feeling... However, the new abi tests were introduced to guard against accidental unintended abi changes. So, it's kind of expected to be annoying if you're intentionally modifying the abi-related code...

Previously, we're really bothered to do manual abi compatibility check every time, even taking into account of deeply-nested structs. I hope this speeds up our development in overall.