fix(storage): fallback lookups for pruned history

[shekhirin]

Changes

Missed cursor jump to delete the original shard

We did a jump to the previous shard

reth/crates/prune/src/pruner.rs

Lines 529 to 530 in aa9036e

	if let Some(prev_value) = cursor
	.prev()?

and if it wasn't found for the same key, we needed to first jump back to the original shard, and only then delete current entry:

reth/crates/prune/src/pruner.rs

Lines 544 to 547 in aa9036e

	// Jump back to the original last shard.
	cursor.next()?;
	// Delete shard.
	cursor.delete_current()?;

HistoryInfo lookups with pruned history

History shard exists

Two problems:

1. There was a missing check

   reth/crates/storage/provider/src/providers/state/historical.rs

   Line 151 in aa9036e

   chunk.select(rank) as u64 != self.block_number &&

   that is required even with unpruned history. rank == 0 can mean not only that the target block is less then the first block in the shard, but also that the first block in shard is equal to the target block, in which case we need to look it up in the changeset.
2. If the target block is not found in any shard and the history is pruned, it doesn't always mean that it's the first write ever. So we need to fallback to the changeset lookup because the key may have been actually written:

   reth/crates/storage/provider/src/providers/state/historical.rs

   Lines 154 to 161 in aa9036e

   	if lowest_available_block_number.is_some() {
   	// The key may have been written, but due to pruning we may not have changesets
   	// and history, so we need to make a changeset lookup.
   	Ok(HistoryInfo::InChangeset(chunk.select(rank) as u64))
   	} else {
   	// The key is written to, but only after our block.
   	Ok(HistoryInfo::NotYetWritten)
   	}

History shard does not exist

If the target block is not found in any shard and the history is pruned, it doesn't always mean that it's the first write ever. So we need to fallback to the plain state lookup because the key may have been actually written:

reth/crates/storage/provider/src/providers/state/historical.rs

Lines 170 to 174 in aa9036e

	} else if lowest_available_block_number.is_some() {
	// The key may have been written, but due to pruning we may not have changesets and
	// history, so we need to make a plain state lookup.
	Ok(HistoryInfo::MaybeInPlainState)
	} else {

[codecov]

Codecov Report

Merging #4121 (4ceff49) into main (fd7e28e) will decrease coverage by 0.02%.
Report is 1 commits behind head on main.
The diff coverage is 94.87%.

Files Changed	Coverage Δ
crates/prune/src/pruner.rs	82.70% <0.00%> (-0.14%)	⬇️
...storage/provider/src/providers/state/historical.rs	87.14% <97.36%> (+0.10%)	⬆️

... and 14 files with indirect coverage changes

Flag	Coverage Δ
integration-tests	16.85% <0.00%> (+<0.01%)	⬆️
unit-tests	64.09% <94.87%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
reth binary	25.94% <ø> (+0.54%)	⬆️
blockchain tree	82.83% <ø> (ø)
pipeline	90.07% <ø> (ø)
storage (db)	74.75% <97.36%> (+0.02%)	⬆️
trie	94.71% <ø> (ø)
txpool	48.23% <ø> (-0.03%)	⬇️
networking	77.69% <ø> (-0.14%)	⬇️
rpc	58.68% <ø> (+<0.01%)	⬆️
consensus	63.76% <ø> (ø)
revm	32.26% <ø> (ø)
payload builder	6.57% <ø> (ø)
primitives	87.88% <ø> (-0.03%)	⬇️

[mattsse]

nice find!