New trie cache causes child_storage_get to panic in PVF

[athei]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

When a cumulus built block is validated which uses child trie storage the PVF panics with the following error:

AbortedDueToTrap(MessageWithBacktrace { message: "wasm trap: wasm `unreachable` instruction executed", backtrace: Some(Backtrace { backtrace_string: "
    0: 0x2c204d - <unknown>!rust_begin_unwind
    1: 0x1e42 - <unknown>!core::panicking::panic_fmt::h41a3520d548dd50f
    2: 0x2736 - <unknown>!core::result::unwrap_failed::hf3fd9d2ae65f7535
    3: 0x2611c8 - <unknown>!<sp_state_machine::ext::Ext<H,B> as sp_externalities::Externalities>::child_storage::h9d038aaae6deb8cd
    4: 0x3079ed - <unknown>!<wasm function 2442>
    5: 0x2583cf - <unknown>!cumulus_pallet_parachain_system::validate_block::implementation::host_default_child_storage_get::haf197c0b2a59ee99
    6: 0x307649 - <unknown>!<wasm function 2418>
    7: 0x1269d5 - <unknown>!pallet_contracts::wasm::runtime::Runtime<E>::get_storage::h7d87f8dc75ae5c45
    8: 0x1136d0 - <unknown>!<pallet_contracts::wasm::runtime::Env as pallet_contracts::wasm::env_def::FunctionImplProvider<E>>::impls::get_storage::h419bbd98f413e490
    9: 0x3043af - <unknown>!<wasm function 2040>
   10: 0x12230d - <unknown>!wasmi::func::FuncInstance::invoke::hb8c52c61278adb2c
   11: 0x122a6a - <unknown>!wasmi::func::FuncInstance::invoke::hb8c52c61278adb2c
   12: 0xb5bb8 - <unknown>!pallet_contracts::exec::Stack<T,E>::run::hdb986f348fffd544
   13: 0x147de9 - <unknown>!<pallet_contracts::exec::Stack<T,E> as pallet_contracts::exec::Ext>::call::ha42a6a96c18df574
   14: 0x1294c5 - <unknown>!pallet_contracts::wasm::runtime::Runtime<E>::call::h11744d3a38b8cee1
   15: 0x114a74 - <unknown>!<pallet_contracts::wasm::runtime::Env as pallet_contracts::wasm::env_def::FunctionImplProvider<E>>::impls::call::h20a671642e6cee5f
   16: 0x304499 - <unknown>!<wasm function 2046>
   17: 0x12230d - <unknown>!wasmi::func::FuncInstance::invoke::hb8c52c61278adb2c
   18: 0x122a6a - <unknown>!wasmi::func::FuncInstance::invoke::hb8c52c61278adb2c
   19: 0xb5bb8 - <unknown>!pallet_contracts::exec::Stack<T,E>::run::hdb986f348fffd544
   20: 0x74086 - <unknown>!pallet_contracts::<impl pallet_contracts::pallet::Pallet<T>>::internal_call::hd6e1c42f86ab766a
   21: 0x532cc - <unknown>!<contracts_rococo_runtime::Call as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::hf6001a29b2ced982
   22: 0x4981a - <unknown>!<contracts_rococo_runtime::Call as sp_runtime::traits::Dispatchable>::dispatch::h79f4ca78f2865beb
   23: 0x21d63d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h43b5552d7c5a1cf0
   24: 0x216950 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::hfa1405b3850c5168
   25: 0x24da22 - <unknown>!cumulus_pallet_parachain_system::validate_block::implementation::validate_block::h6da71c81c4188f23
   26: 0x246441 - <unknown>!validate_block
   27: 0x307470 - <unknown>!<wasm function 2405>
" }) })")

Please note:

• It does not happen when you set --trie-cache-size 0 on the collator.
• The panic appears in the INFO log level on the validator so watch out for that

Steps to reproduce

It is unclear what the minimal reproducer it but we noticed when using pallet-contracts on a rococo parachain. Since then we added --trie-cache-size 0 to the collators of that parachain. Hence a local testnet is needed to reproduce. In the following I describe the steps which we were able to reproduce the error. It did not work 100% but just doing this once but needed some transactions to be submitted.

You will reproduce the error deploying and calling some smart contracts on a parachain. Easiest way to interact with the parachain is to use https://contracts-ui.substrate.io/ and select "Local Node" in the drop down.

1. You need the following two contracts. Download those files (you will upload them to the chain later). Those are pretty big because they are built with debug information: contracts.zip
2. Deploy game.contract and set the params of the constructor like this:

3. Deploy `player.contract` and set the constructor parameter to the game contract (deployed in 2.). You will be able to select it from the drop down:

4. Now call `game::submit_turn(player_contract)` until you observe the error on the validator. This will essentially stall the parachain as the collator will always retry to include this invalid transaction:

Please keep in mind when submitting a turn: The game does only allow you to submit one turn per block. The dry-run doesn't refresh automatically so you would need to select another message and go back to make sure it runs against the current block. Otherwise it might be stale.