v2.1: Blockstore: Migrate ShredIndex type to more efficient data structure (backport of #3900)

[mergify]

Problem

The current blockstore index type, backed by a BTreeSet, suffers from performance issues in serialization / deserialization due to its dynamically allocated and balanced nature. See #3570 for context.

Summary of Changes

Fixes #3570

This PR implements the ShredIndex behavior behind a bit vec (Vec<u8>).

Migration strategy

The general goal is to avoid the overhead of writing two separate columns while still supporting the downgrade path. While this can be accomplished with two distinct columns, for this proposal, rather than writing to a new column, I am proposing we add support for both formats in the same column. This will avoid an additional read to rocksdb, as we can attempt deserialization of both formats on the same rocksdb slice. In order to support the downgrade path, this initial PR will solely add support for reading the new format.

See release steps

The idea here is to split the column migration across three releases such that:

1. Initial release simply adds support for reading the new format as a fallback case, and does no writing of the new format.
   • This lays the foundation for a downgrade. For example, assume operators have upgraded to release 2/3 in the chain (bullet point 2), and as such have been solely writing to the new format. In the event of a downgrade, release 1/3 still understands how to read the new format, while continuing to read and write the legacy version.
   • This ensures release 1/3 doesn't incur the overhead of serializing and writing the new format, but can still understand and use it in the event of a downgrade.
2. This release reads and writes the new format as its primary target, yet still understands the legacy column for fallback reads (i.e., we swap the deserialization attempt order). It does no writing of the legacy format.
   • This instantiates the migration. We can safely downgrade to release 1 because it understands how to read the new format that was written in release 2.
3. Once the release is considered stable and we don't anticipate a downgrade, we can remove support for the legacy format and its associated fallback reads.

---

This is an automatic backport of pull request #3900 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of f8e5b16 has failed:

On branch mergify/bp/v2.1/pr-3900
Your branch is up to date with 'origin/v2.1'.

You are currently cherry-picking commit f8e5b1672.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   Cargo.lock
	modified:   ledger/Cargo.toml
	new file:   ledger/proptest-regressions/blockstore_meta.txt
	modified:   ledger/src/blockstore_meta.rs
	modified:   programs/sbf/Cargo.lock

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   ledger/src/blockstore_db.rs
	deleted by us:   svm/examples/Cargo.lock

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[t-nelson]

svm/examples/Cargo.lock was erroneously introduced here?

[t-nelson]

what's the motivation for backporting here? we'll very likely be beginning voluntary mb adoption of 2.1 this week, so the bar for backport with little/no bake time is exceptionally high

[cpubot]

svm/examples/Cargo.lock was erroneously introduced here?

Oh, yeah, you're right. Didn't realize that file doesn't exist in 2.1. Let me remove it.

[cpubot]

what's the motivation for backporting here? we'll very likely be beginning voluntary mb adoption of 2.1 this week, so the bar for backport with little/no bake time is exceptionally high

The motivation here is that this PR just sets up a rollback strategy for the new ShredIndex type that will be introduced in the next release. From the description:

The idea here is to split the column migration across three releases such that:

1. Initial release simply adds support for reading the new format as a fallback case, and does no writing of the new format.
   - This lays the foundation for a downgrade. For example, assume operators have upgraded to release 2/3 in the chain (bullet point 2), and as such have been solely writing to the new format. In the event of a downgrade, release 1/3 still understands how to read the new format, while continuing to read and write the legacy version.
   - This ensures release 1/3 doesn't incur the overhead of serializing and writing the new format, but can still understand and use it in the event of a downgrade.
2. This release reads and writes the new format as its primary target, yet still understands the legacy column for fallback reads (i.e., we swap the deserialization attempt order). It does no writing of the legacy format.
   - This instantiates the migration. We can safely downgrade to release 1 because it understands how to read the new format that was written in release 2.
3. Once the release is considered stable and we don't anticipate a downgrade, we can remove support for the legacy format and its associated fallback reads.

In short, this PR just enables deserialization of the new blockstore index type (IndexV2) into the current blockstore index type (Index). If IndexV2 is enabled as the primary format in 2.2, for example, we don't have to worry about a rollback to 2.1 not being able to deserialize the new format. The new format sees a roughly 2x speed up of the insert_shreds_elapsed_us metric (see here), and we'd like to roll that out in 2.2.

[alessandrod]

what's the motivation for backporting here? we'll very likely be beginning voluntary mb adoption of 2.1 this week, so the bar for backport with little/no bake time is exceptionally high

To reiterate, the PR doesn't change any behavior. It makes 2.1 capable of reading 2.2 ledgers which is when we'll change format.

[steviez]

Building on top of what Zach and Alessandro mentioned, this is where there is "new behavior" in production.

• We read from Blockstore and try to deserialize into Index (the pre-existing type). If deserialize fails ...
  • Existing v2.1 and older code will panic if deserialize into Index fails (the caller unwraps this):
    

    agave/ledger/src/blockstore_db.rs

    Lines 1730 to 1733 in 33ab693

    	if let Some(pinnable_slice) = self.backend.get_pinned_cf(self.handle(), key)? {
    	let value = deserialize(pinnable_slice.as_ref())?;
    	result = Ok(Some(value))
    	}

  • This PR will instead try to deserialize into IndexV2 ONLY if initial deserialize fails. But, seeing as we aren't writing the new format, that deserialize will fail too