Penumbra support

[agouin]

Working branch for penumbra integration

[conorsch]

Took this for a spin. As shown in the CI error for the build step, I'm unable to build the binary locally, but I was able to fetch the container image at ghcr.io/cosmos/relayer:penumbra and pilfer the rly binary from there. That's what I used to test locally.

Following the relayer setup guide:

1. ✔️ rly config init
2. ✔️ rly chain add (after templating out accurate chain_id info from the two Penumbra testnets)
3. 🟧 rly keys add ; this command emits a valid seed phrase, but the penumbra addr displayed is invalid; that's OK for now, since Penumbra doesn't have fees yet, so we'll just ignore the malformed addr for now
4. ✔️ rly paths new linking the two testnet chains
5. ❌ rly transact link fails with:

getting penumbra trusting period
getting penumbra trusting period
got data of type cosmos.CosmosMessage but wanted provider.CosmosMessage 
success
Error: error creating clients: failed to create client on dst chain{penumbra-testnet-carme}: client identifier event attribute not found

From that message I believe there's more implementation work required in the go code before we can relay events. The complete script I used to test can be found in penumbra-zone/penumbra#2166.

[conorsch]

Running make test and make build locally now results in a working rly binary. Thanks for fixing the build!

To test functionality, I resumed attempts to build a path between two Penumbra test chains, via rly transact link penumbra_chain. The process failed at the rly transact clients penumbra_chain step, which is the first of three steps within the link action, according to the cross-chain docs.

full error logs from rly

2023-03-22T19:07:58.090363Z     info    waiting for penumbra tx to commit       {"provider_type": "penumbra"}
2023-03-22T19:07:58.094010Z     info    waiting for penumbra tx to commit       {"provider_type": "penumbra"}
2023-03-22T19:07:58.135711Z     info    Error building or broadcasting transaction      {"provider_type": "penumbra", "chain_id": "penumbra-testnet-carme-e248c323", "attempt": 1, "max_attempts": 5, "error": "RPC error -32603 - Internal error: tx (05F5A6E226309A2641B6B260387B4D5A53A9C1B4A21B375BD333A94CC64AEECD) not found"}
2023-03-22T19:07:58.159496Z     info    Error building or broadcasting transaction      {"provider_type": "penumbra", "chain_id": "penumbra-testnet-carme", "attempt": 1, "max_attempts": 5, "error": "RPC error -32603 - Internal error: tx (426ACA2BEBF44FEB59B0EA542FBCCC9730F31048BCD9C08666EA62F522568E37) not found"}
2023-03-22T19:07:58.619997Z     info    Error building or broadcasting transaction      {"provider_type": "penumbra", "chain_id": "penumbra-testnet-carme-e248c323", "attempt": 2, "max_attempts": 5, "error": "RPC error -32603 - Internal error: tx (05F5A6E226309A2641B6B260387B4D5A53A9C1B4A21B375BD333A94CC64AEECD) not found"}
2023-03-22T19:07:58.689258Z     info    Error building or broadcasting transaction      {"provider_type": "penumbra", "chain_id": "penumbra-testnet-carme", "attempt": 2, "max_attempts": 5, "error": "RPC error -32603 - Internal error: tx (426ACA2BEBF44FEB59B0EA542FBCCC9730F31048BCD9C08666EA62F522568E37) not found"}
2023-03-22T19:07:59.556380Z     info    Error building or broadcasting transaction      {"provider_type": "penumbra", "chain_id": "penumbra-testnet-carme", "attempt": 3, "max_attempts": 5, "error": "RPC error -32603 - Internal error: tx (426ACA2BEBF44FEB59B0EA542FBCCC9730F31048BCD9C08666EA62F522568E37) not found"}
success
2023-03-22T19:07:59.558245Z     info    Failed sending transaction      {"chain_id": "penumbra-testnet-carme", "msg-0": {"msg_json": "{\"client_state\":{\"chain_id\":\"penumbra-testnet-carme-e248c323\",\"trust_level\":{\"numerator\":\"1\",\"denominator\":\"3\"},\"trusting_period\":\"7200000000000\",\"unbonding_period\":\"14400000000000\",\"max_clock_drift\":\"600000000000\",\"frozen_height\":{},\"latest_height\":{\"revision_height\":\"14439\"},\"proof_specs\":[{\"leaf_spec\":{\"hash\":1,\"prehash_key\":1,\"prehash_value\":1,\"prefix\":\"Sk1UOjpMZWFmTm9kZQ==\"},\"inner_spec\":{\"child_order\":[0,1],\"child_size\":32,\"min_prefix_length\":16,\"max_prefix_length\":48,\"hash\":1},\"max_depth\":64},{\"leaf_spec\":{\"hash\":1},\"inner_spec\":{\"child_order\":[0,1],\"child_size\":32,\"hash\":1},\"max_depth\":1}],\"upgrade_path\":[\"upgrade\",\"upgradedIBCState\"],\"allow_update_after_expiry\":true,\"allow_update_after_misbehaviour\":true},\"consensus_state\":{\"timestamp\":\"2023-03-22T19:07:48.270361507Z\",\"root\":{\"hash\":\"N0SzRdp2Wal0Hsth5wap4NjNoaODj6RCd4rwLqyVH4k=\"},\"next_validators_hash\":\"ADDD487E363A01400DA36C18E8E7234F39F64406BA508905FD97BA4A026516B1\"},\"signer\":\"penumbrav2t189eghyul0ct0fdfhlgdqerrd3n8ny8a456pgcg\"}"}, "error": "context canceled"}
Error: failed to create client on src chain{penumbra-testnet-carme-e248c323}: client identifier event attribute not found

Despite that pessimistic error message, the create_client transaction identified by 426ACA2BEBF44FEB59B0EA542FBCCC9730F31048BCD9C08666EA62F522568E37 is indeed visible on the chain:
https://rpc.testnet.penumbra.zone/tx?hash=0x426ACA2BEBF44FEB59B0EA542FBCCC9730F31048BCD9C08666EA62F522568E37

For completeness's sake, I can query that tx hash via pcli as well:

❯ pcli q tx 426ACA2BEBF44FEB59B0EA542FBCCC9730F31048BCD9C08666EA62F522568E37
Scanning blocks from last sync height 25510 to latest height 117171
[3s] ██████████████████████████████████████████████████   91661/91661   25207/s ETA: 0s
{
  "body": {
    "actions": [
      {
        "ibcAction": {
          "rawAction": {
            "typeUrl": "/ibc.core.client.v1.MsgCreateClient",
            "value": "CscBCisvaWJjLmxpZ2h0Y2xpZW50cy50ZW5kZXJtaW50LnYxLkNsaWVudFN0YXRlEpcBCh9wZW51bWJyYS10ZXN0bmV0LWNhcm1lLWUyNDhjMzIzEgQIARADGgMIoDgiAwjAcCoDCNgEMgA6AxDncEInChUIARABGAEqDUpNVDo6TGVhZk5vZGUSDAoCAAEQIBgQIDAwARhAQhAKAggBEggKAgABECAwARgBSgd1cGdyYWRlShB1cGdyYWRlZElCQ1N0YXRlUAFYARKGAQouL2liYy5saWdodGNsaWVudHMudGVuZGVybWludC52MS5Db25zZW5zdXNTdGF0ZRJUCgwIhKTtoAYQo8f1gAESIgogN0SzRdp2Wal0Hsth5wap4NjNoaODj6RCd4rwLqyVH4kaIK3dSH42OgFADaNsGOjnI0859kQGulCJBf2XukoCZRaxGjJwZW51bWJyYXYydDE4OWVnaHl1bDBjdDBmZGZobGdkcWVycmQzbjhueThhNDU2cGdjZw=="
          }
        }
      }
    ],
    "fee": {
      "amount": {}
    }
  },
  "bindingSig": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==",
  "anchor": {
    "inner": "rNiRHp+gMUoDk1wALj4uBmkxyBnlCY097HS793TKeQw="
  }
}

On the backend, pd logs show that the client registrations are recognized. Note the suffix integer, e.g. 07-tendermint-19, which shows how many times I've run the command. =)

2023-03-22T19:07:58.070773Z  INFO abci:CheckTx{kind=New txid="426aca2bebf44feb59b0ea542fbccc9730f31048bcd9c08666ea62f522568e37"}:execute:IbcAction{idx=0}: deserializing client state
2023-03-22T19:07:58.070952Z  INFO abci:CheckTx{kind=New txid="426aca2bebf44feb59b0ea542fbccc9730f31048bcd9c08666ea62f522568e37"}:execute:IbcAction{idx=0}: creating client ClientId("07-tendermint-19")
2023-03-22T19:07:58.071481Z  INFO abci:CheckTx{kind=New txid="426aca2bebf44feb59b0ea542fbccc9730f31048bcd9c08666ea62f522568e37"}: tx accepted elapsed=1.65492ms
2023-03-22T19:08:00.851087Z  INFO abci:DeliverTx{txid="426aca2bebf44feb59b0ea542fbccc9730f31048bcd9c08666ea62f522568e37"}:execute:IbcAction{idx=0}: deserializing client state
2023-03-22T19:08:00.851715Z  INFO abci:DeliverTx{txid="426aca2bebf44feb59b0ea542fbccc9730f31048bcd9c08666ea62f522568e37"}:execute:IbcAction{idx=0}: creating client ClientId("07-tendermint-19")
2023-03-22T19:08:00.853530Z  INFO abci:DeliverTx{txid="426aca2bebf44feb59b0ea542fbccc9730f31048bcd9c08666ea62f522568e37"}:event{kind="create_client"}: k="client_id" v="07-tendermint-19"
2023-03-22T19:08:00.853550Z  INFO abci:DeliverTx{txid="426aca2bebf44feb59b0ea542fbccc9730f31048bcd9c08666ea62f522568e37"}:event{kind="create_client"}: k="client_type" v="ClientType(07-tendermint)"
2023-03-22T19:08:00.853557Z  INFO abci:DeliverTx{txid="426aca2bebf44feb59b0ea542fbccc9730f31048bcd9c08666ea62f522568e37"}:event{kind="create_client"}: k="consensus_height" v="0-14439"

Based on this information, it looks to me that the rly logic isn't able to query the chain state accurately: the create_client event is successfully posted to the chain, but rly cannot find it later, and so reports a failure. On subsequent runs, rly isn't able to find a previous client registration, and so tries to create one again. Again, the configs I'm using to test can be found in penumbra-zone/penumbra#2166.

[hdevalence]

I was able to run it and pushed it a bit further.

On https://gist.github.com/hdevalence/6317929e8db2b76e429b8a9553b2fecc , I noticed the transaction was spotted but the events were not detected.

Instrumenting them with Printf statements gave

events: [{EventType:create_client Attributes:map[Y29uc2Vuc3VzX2hlaWdodA==:MC0xNjI1MA== Y2xpZW50X2lk:MDctdGVuZGVybWludC0yMg== Y2xpZW50X3R5cGU=:Q2xpZW50VHlwZSgwNy10ZW5kZXJtaW50KQ==]}]
2023-03-22T21:48:49.331578Z    info    Failed sending transaction    {"chain_id": "penumbra-testnet-carme-e248c323", "error": "context canceled"}
Error: error creating clients: failed to create client on dst chain{penumbra-testnet-carme}: client identifier event attribute not found

So, for some reason the event attributes are being base64'd. Why?

I'm not sure that this is correct (I think it's not), but in the meantime I patched the relayer to unbase64 the events in this branch: https://github.com/penumbra-zone/relayer/tree/penumbra-unbase64

After running it again, I eventually got https://gist.github.com/hdevalence/156cb4e3784cfbd96b72087f1ee8f703

which panics on "implement me". There are two unimplemented methods:

func (cc *PenumbraProvider) MsgSubmitQueryResponse(chainID string, queryID provider.ClientICQQueryID, proof provider.ICQProof) (provider.RelayerMessage, error) {
	//TODO implement me
	panic("implement me")
}

func (cc *PenumbraProvider) SendMessagesToMempool(ctx context.Context, msgs []provider.RelayerMessage, memo string, asyncCtx context.Context, asyncCallback func(*provider.RelayerTxResponse, error)) error {
	//TODO implement me
	panic("implement me")
}

The SendMessagesToMempool should work like SendMessages, except that it doesn't need to wait for block inclusion. I think we can leave the other one unimplemented, since we don't support ICQ currently.

For SendMessagesToMempool, we may want to extract a common helper method from SendMessages. The comment in the current SendMessages impl about only one action per tx is not correct, so we should fix that too.

[hdevalence]

Update: I filled in a stub SendMessagesToMempool with a common helper function, but it still doesn't work. The code is in the same penumbra-unbase64 branch.

[agouin]

continued in #1144