State export and genesis migration fails from v0.39.1 to v0.40.0-rc1

[kaustubhkapatral]

Summary of Bug

Chain crashes with the following error after exporting state and migrating genesis from v0.39.1 to v0.40.0-rc1

I[2020-11-02|09:58:25.888] starting ABCI with Tendermint                module=main
panic: proto: Int64Value: illegal tag 0 (wire type 1)

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/store/rootmulti.getLatestVersion(0x202cb60, 0xc000e5a570, 0x2015f40)
        github.com/cosmos/cosmos-sdk@v0.40.0-rc1/store/rootmulti/store.go:903 +0x10b
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).LoadLatestVersion(0xc000f274a0, 0x2015f40, 0x2ddd480)
        github.com/cosmos/cosmos-sdk@v0.40.0-rc1/store/rootmulti/store.go:158 +0x37
github.com/cosmos/cosmos-sdk/baseapp.DefaultStoreLoader(0x2033ec0, 0xc000f274a0, 0xc00011f830, 0xc0004b97e0)
        github.com/cosmos/cosmos-sdk@v0.40.0-rc1/baseapp/baseapp.go:258 +0x31
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).LoadLatestVersion(0xc000129520, 0xc000ffd7f0, 0x7fb864542620)
        github.com/cosmos/cosmos-sdk@v0.40.0-rc1/baseapp/baseapp.go:248 +0x42
github.com/ovrclk/akash/app.NewApp(0x2015780, 0xc0004fefa0, 0x202cb60, 0xc000e5a570, 0x0, 0x0, 0x0, 0xc000135140, 0xc000e475b0, 0xd, ...)
        github.com/ovrclk/akash/app/app.go:407 +0x6aba
github.com/ovrclk/akash/cmd/akashd/cmd.newApp(0x2015780, 0xc0004fefa0, 0x202cb60, 0xc000e5a570, 0x0, 0x0, 0x1fd70c0, 0xc000f78000, 0x30ffffffff, 0x4)  
        github.com/ovrclk/akash/cmd/akashd/cmd/root.go:140 +0x93d
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc000e817e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2015300, 0xc000e4b7c0, ...)
        github.com/cosmos/cosmos-sdk@v0.40.0-rc1/server/start.go:208 +0x168
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000f24840, 0x2e0dbd8, 0x0, 0x0, 0x0, 0x0)
        github.com/cosmos/cosmos-sdk@v0.40.0-rc1/server/start.go:117 +0x154
github.com/spf13/cobra.(*Command).execute(0xc000f24840, 0x2e0dbd8, 0x0, 0x0, 0xc000f24840, 0x2e0dbd8)
        github.com/spf13/cobra@v1.1.0/command.go:850 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000d39600, 0x1bb47f4, 0x5, 0xc000e475b0)
        github.com/spf13/cobra@v1.1.0/command.go:958 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.1.0/command.go:895
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        github.com/spf13/cobra@v1.1.0/command.go:888
github.com/ovrclk/akash/cmd/akashd/cmd.Execute(0xc000d39600, 0x20214a0, 0xc000e60880)
        github.com/ovrclk/akash/cmd/akashd/cmd/root.go:86 +0x165
main.main()
        github.com/ovrclk/akash/cmd/akashd/main.go:12 +0x2a

Version

State export version:
Cosmos SDK version: v0.39.1
Akash version: v0.8.1

Genesis migration version:
Cosmos SDK version: v0.40.0-rc1
Branch name: bigbang

Steps to Reproduce

Exporting state from v0.8.1 binary:

akashd export --for-zero-height  > bigbang_genesis_export.json

Building the bigbang version of akashd and executing the following:

akashd migrate v0.40 bigbang_genesis_export.json > new_v40_genesis.json

Copying the new genesis to config directory and starting the chain:

cp new_v40_genesis.json ~/.akashd/config/genesis.json

akashd start

Folder with original genesis, exported state and migrated genesis: issue.tar.gz

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[robert-zaremba]

Note: It seams that the same commands worked on Gaia. I was able to load exported gaia snapshot. AFAIK @zmanian did the whole sequence (export -> migrate -> init -> start) successful (though it took lot of time because of #7682)

[robert-zaremba]

Also, we have rc2 over the corner - I would suggest debugging and redoing using rc2. What do you think?
/cc @clevinson

[zmanian]

You need to provide an implementation of the tendermint conversion.

This is done here.
https://github.com/cosmos/gaia/blob/jack/gaiav3.0/app/migrate.go#L198-L229

[clevinson]

rc2 is now tagged if you'd like to test again against that.

Zaki is correct in that you do need to provide a manual migration for the tendermint consesnsus params. That needs to happen either as part of an application migrate script, or externally via jq manually (see Step #7 here). I'm unsure if the error reported here is corresponding to tm consensus params, but they need to be manually migrated either way, so make sure that is part of your migration steps.

PS- @anilcse let's make sure we include the requirement about migrating tendermint consensus params in the migration guide docs you're working on

[amaury1093]

In the attached issue.tar.gz, the genesis file looks good (tendermint conversion has already been done correctly).

However, I couldn't reproduce the panic with the attached new_v40_genesis.json, the chain ran fine. Would like more info.

[anilcse]

You need to provide an implementation of the tendermint conversion.

This is done here.
https://github.com/cosmos/gaia/blob/jack/gaiav3.0/app/migrate.go#L198-L229

Thanks @zmanian , looks like these changes are covered in migrate command already.

@amaurymartiny for bigbang-1 testnet upgrade failure, looks like it is an issue with upgrade-handler, it didn't handle the state migrations (still debugging on this). For migrate + upgrade, it was just an issue with config files. We needed to update the config files and it's working!

Planning an internal testnet tomorrow to cross check this upgrade path and will close this issue once after the testing.

[zmanian]

Your migration command also need to provide intitialization state for new modules if you are upgrading like IBC.

[robert-zaremba]

Fixed by @amaurymartiny