State reconstruction broken on Holesky due to backfill bug for skipped slots

[michaelsproul]

Description

There's a bug in block backfill here:

lighthouse/beacon_node/beacon_chain/src/historical_blocks.rs

Lines 122 to 134 in 441fc16

	// If we've reached genesis, add the genesis block root to the batch and set the
	// anchor slot to 0 to indicate completion.
	if expected_block_root == self.genesis_block_root {
	let genesis_slot = self.spec.genesis_slot;
	chunk_writer.set(
	genesis_slot.as_usize(),
	self.genesis_block_root,
	&mut cold_batch,
	)?;
	prev_block_slot = genesis_slot;
	expected_block_root = Hash256::zero();
	break;
	}

If there are skipped slots between genesis and the first block, then we will never write a block root for these slots. This breaks the forwards block roots iterator, and consequently state reconstruction. This is exactly what happens on Holesky, because slot 1 is skipped.

Version

Lighthouse v4.5.0 (and all previous versions). Observable on v4.5.0 because of Holesky. Tree-states is also broken (it actually panics).

Steps to resolve

• We should fill in the intermediate block roots when storing the genesis block root.

Workarounds

• Users that need state reconstruction on Holesky right now should re-sync from genesis (without checkpoint sync).
• Once a fix is available users can do a re-sync using checkpoint sync. Existing Holesky databases that were backfilled to genesis are corrupt and should be deleted. There is no plan to provide a fix for these.

[michaelsproul]

This is fixed on tree-states as of v4.5.444-exp.

[michaelsproul]

Fixed on unstable. Will be in 4.6