Rocksdb Error when Downgrading from Besu 23.7.1 to 22.10.2

[sheharyar12]

Rocksdb Error when Downgrading from Besu 23.7.1 to 22.10.2

• [ x] Have you read the Code of Conduct? By filing an Issue, I agree to comply with it, including treating everyone with respect: Code of Conduct
• [ x] Reproduced the issue in the latest version of the software
• [ x] Read the debugging docs: Besu Debugging Docs
• [ x] Checked for duplicate issues: Besu Issues

Description

I want to seamlessly revert from version 23.7.1 to 22.10.2 without any database errors after having successfully upgraded to 23.7.1 and executed the recommended database revert command, and finally downgrading back to 22.10.2.

Acceptance Criteria

Besu should revert back to version 22.10.2 without any startup errors after previously upgrading to 23.7.1 and executing the recommended database revert commands. See here as recommended.

Steps to Reproduce (Bug)

1. Deploy Besu 22.10.2 (with 4 min validators) as a private network.
2. Allow blocks to be imported.
3. Stop Besu 22.10.2 and run besu with version 23.7.1.
4. Allow it to import blocks on 23.7.1.
5. Stop the 23.7.1 version and execute the command: besu revert storage-variables
6. Redeploy using version 22.10.2. An error should be encountered during startup.

Expected behavior: Besu 22.10.2 should start without errors after reverting from 23.7.1.

Actual behavior: Encountered an error during the startup of Besu 22.10.2 after reverting.

Frequency: This issue occurs consistently upon reverting.

Logs when reverting db via executing subcommand storage revert-variables

2023-08-18 14:29:43.219+00:00 | main | INFO | DatabaseMetadata | Lookup database metadata file in data directory: /besu/data
2023-08-18 14:29:43.835+00:00 | main | INFO | RocksDBKeyValueStorageFactory | Existing database detected at /besu/data. Version 1. Compacting database..
2023-08-18 14:29:52.733+00:00 | main | INFO | StorageSubCommand$Revert VariablesStorage | Reverted variable storage for key chainHeadHash
2023-08-18 14:29:52.821+00:00 | main | INFO | StorageSubCommand$RevertVariablesStorage | Reverted variable storage for key local-enr-seqno

Logs (if a bug)

2023-08-22 00:11:15.041+00:00 I main I INFO I RocksDBKeyValueStorageFactory | Existing database detected at /besu/data. Version 1
picocli.CommandLine$ParameterException: org.rocksdb.RocksDBException: Column families not opened: O
at org.hyperledger.besu.cli.BesuCommand.run(BesuCommand.java:1443)
at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
at picocli.CommandLine.execute(CommandLine.java:2078)
at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:54)
at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:32)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
at picocli.CommandLine.execute(CommandLine.java:2078)
at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:1614)
at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:1391)
at org.hyperledger.besu.Besu.main(Besu.java:48)
Caused by: picocli.CommandLine$ExecutionException: org.rocksdb.RocksDBException: Column families not opened: O
at org.hyperledger.besu.cli.BesuCommand.buildController(BesuCommand.java:2122)
at org.hyperledger.besu.cli.BesuCommand.initController(BesuCommand.java:2115)
at org.hyperledger.besu.cli.BesuCommand.run(BesuCommand.java:1427)
... 16 more
Caused by: org.hyperledger.besu.plugin.services.exception.StorageException: org.rocksdb.RocksDBException: Column families not opened: D
at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueStorage.<init>(RocksDBColumnarKeyValueStorage.java:166)
at org.hyperledger.besu.plugin.services.storage.rocksdb.RocksDBKeyValueStorageFactory.create(RocksDBKeyValueStorageFactory.java:124)
at org.hyperledger.besu.ethereum.storage.keyvalue.KeyValueStorageProviderBuilder.build(KeyValueStorageProviderBuilder.java:71)
at org.hyperledger.besu.cli.BesuCommand.keyValueStorageProvider(BesuCommand.java:2803)
at org.hyperledger.besu.cli.BesuCommand.getControllerBuilder(BesuCommand.java:2127)
at org.hyperledger.besu.cli.BesuCommand.buildController(BesuCommand.java:2120)
Caused by: org.rocksdb.RocksDBException: Column families not opened: [
at org.rocksdb.OptimisticTransactionDB.open(Native Method)
at org.rocksdb.OptimisticTransactionDB.open(OptimisticTransactionDB.java:80)
at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueStorage.<init>(RocksDBColumnarKeyValueStorage.java:146)
... 23 more
To display full help: besu [COMMAND] --help

Software version: Initially besu 22.10.2 then upgraded to 23.7.1. Finally wanted to revert back to 22.10.2

Additional Information

Database storage format : FOREST

[non-fungible-nelson]

Hi @sheharyar12 thanks for the report.

@garyschulte can you take a look in fabio's absence?

[garyschulte]

I was able to reproduce this locally. storage revert-variables works correctly.

However a breaking change was introduced in 23.1.0 that added a new column family for "chain pruner state". What 22.10.2 is complaining about isn't the storage varaibles, it is complaining that it doesn't know about the new chain pruner column family.

This error Column families not opened: O is beyond useless, I will put up a PR to fix this error message to be more useful by translating the column family index byte to the column family name.

If you need to revert these databases to 22.10.2, reach out on the hyperledger discord and we can help you manually revert

[garyschulte]

#4686 should only add the chain pruning column family if it is explicitly enabled. It isn't clear why the chain pruning column family is being created when running 23.7.1. I will dig into this a bit more and see where/why this is occurring.

[garyschulte]

Found, fixed in #5830. Sorry for the inconvenience. Again, if you need help migrating those databases back to 22.10.2 reach out on discord and we can help.

[sheharyar12]

Quick fix! Thank you all so much ! @garyschulte @non-fungible-nelson