State Chain Mock

[kylezs]

Wanted to raise this as a discussion to collect ideas on how to approach this.

Currently testing the CFE is difficult because what it does and what states it gets into is entirely dependent on the State Chain and the events it emits.

Testing the frontend is equally difficult, we depend on a running state chain, and when the state chain crashes, we can no longer test frontend features.

What I think a mock could look like:

• Full JsonRPC interface: https://github.com/paritytech/jsonrpc -> This way we are testing all the code in the CFE, including it's RPC code. We can use traits that exist on the SC to ensure parity and type safety with the real SC.
• Each test can instantiate its own blocks on the State Chain. Let's say I want to test that a Keygen, I could do that by creating a block that contains the KeygenRequest event, then start the sc_observer and ensure it sends the Keygen instruction through the channel.
• We could have a test for the frontend, that simply creates a SC block that contains every possible event (we can use the root Event enumeration to ensure it’s type safe / exhaustive) and then in order to test we have the types correct, we would only have to try and index that one block. We wouldn’t have to create all these events by rotating, claiming etc. We can just insert them manually into the block to be read

[cdrn]

I can chime in here. This would be enormously useful for us. Currently most of the errors we're experiencing with the subsquid indexer involve bad types which are not exposed to polkadotJS until the relevant event is emitted from the chain. When the badly typed/untyped event is emitted, it can crash the indexer. I take it by generating a block with every possible event, we would be able to see these potential type issues before we go to prod. If so, big +1.

[dandanlen]

I've said this before but I would rather ensure that we can test things in isolation. If we can't, then we should refactor so that we can.

If we have N different subsystems each with M different states, an integration test needs to cover all possible combinations: M^N possible states. This escalates quickly - it becomes impossible to know what is and isn't covered because you have to trace all possible code paths to know what states the system can reach.

If we test in isolation, we have M*N states to test.

The type checking is a good example: it does not rely in any way on the rpc interface, or building a state chain block, or subscription channels etc. The way to do this is to write a test that takes a serialized event, and tries to deserialize it based on the available metadata. Then we can generate each possible event, serialize it, and run the deserialization test against each event.

I think this is more or less what you are proposing, I'm just saying we should not couple this to testing a bunch of other stuff like rpcs or block authorship.

For example:

• Each test can instantiate its own blocks on the State Chain. Let's say I want to test that a Keygen, I could do that by creating a block that contains the KeygenRequest event, then start the sc_observer and ensure it sends the Keygen instruction through the channel.

If we can do this (instantiate a block) then we don't need to put it on a running network in order to test it via rpc.

[andyjsbell]

We should also consider the maintenance and debt we would be creating potentially with a batch of tests which seem to confirming already things known(and well tested) and not testing, as Dan says, in isolation. As our communication between the sub-components(SC, CFE, FE(Indexer)) hinge on events then maybe we should be looking at something which is lighter in terms of mocking. The integration tests for the SC have a simply mock the CFE with the interaction of events and calls across the two.

[kylezs]

Just so this conversation is wrapped up, we decided to go with use a trait to wrap the RPCs that connect to external sources so we can use mockall to mock the responses of any requests made to the state chain (and an ethereum node). See #1130

[kylezs]

This isn't helpful for the frontend team though