bug(trie): invalid trie masks

[rkrasiuk]

Description

Currently, we commit intermediate trie nodes with invalid trie masks (issue to be amended with the cause after investigation cc @fgimenez). This causes trie walker to have invalid stack when it calls self.consume_node and pushes a subnode to the stack which does not share a common prefix with the previous last node on the stack.

reth/crates/trie/trie/src/walker.rs

Lines 135 to 148 in 8605d04

	if !self.can_skip_current_node && self.children_are_in_trie() {
	// If we can't skip the current node and the children are in the trie,
	// either consume the next node or move to the next sibling.
	match last.nibble() {
	-1 => self.move_to_next_sibling(true)?,
	_ => self.consume_node()?,
	}
	} else {
	// If we can skip the current node, move to the next sibling.
	self.move_to_next_sibling(false)?;
	}
	// Update the skip node flag based on the new position in the trie.
	self.update_skip_node();

Once the subnode is done processing, it comes back to a previous one and returns a node key that is out of order causing has builder to panic.

Addition Info

Issue reports: #11471, #11955

[DaveWK]

I saw the recent fix come in and was wondering if this also has the potential to write a bunch of dangling storage tries that never get used?

I noticed there's the command reth recover storage-tries and wondering if this would be beneficial to run on my node to potentially clean up bad entries that are being written/read from disk and impacting the db performance. Also wondering if this storage trie cleanup ran periodically during normal operation as a form of maintenance/vacuuming to reduce excess writes/storage.

[fgimenez]

We have tests that show the issue here #12080