Add pab-local-cluster

[j-mueller]

• Add a pab-local-cluster tool that starts up a local cluster (private testnet) with wallet, chain index, PAB
  • Parts of it are copied from cardano-wallet
• Split up the chain index project into
  • plutus-chain-index-core: The chain index libraries (used to be the plutus-chain-index project)
  • plutus-chain-index: A new library that implements the executable / app. This was done so that we can import the code in other places, which wasn't possible when it was an executable.

Running the integration tests

Private testnet (local cluster)

1. cabal build plutus-pab-local-cluster cardano-node cardano-wallet
2. Clone https://github.com/input-output-hk/cardano-wallet/ and set the SHELLEY_TEST_DATA environment variable to the absolute path of the plutus-pab/local-cluster/cluster-data/cardano-node-shelley in your local clone of the plutus-apps repository.
3. cabal exec plutus-pab-local-cluster
4. Wait until the message Starting PAB backend server on port 9080 appears
5. Run curl -H "Content-Type: application/json" -v -X POST -d '{"caID":{"tag":"IntegrationTest"},"caWallet":{"getWalletId":"2d4cc31a4b3116ab86bfe529d30d9c362acd0b44"}}' localhost:9080/api/contract/activate to start the test. A couple of log messages appear, the last one should say something like fbca6236-f532-43d9-89db-6ddc7002336e: "Tx confirmed. Integration test complete."

Public testnet

Assuming you have checked out the plutus apps repository in ${plutus-apps}

1. cabal build plutus-pab-examples plutus-chain-index cardano-node cardano-wallet
2. Run the ${plutus-apps}/plutus-pab/test-node/start-testnet-node.sh script, wait for cardano node to synchronise
3. Run cabal exec -- cardano-wallet serve --testnet ${plutus-apps}/plutus-pab/test-node/testnet/testnet-byron-genesis.json --node-socket ${plutus-apps}/plutus-pab/test-node/testnet/node.sock in a new terminal to start the wallet
4. Create or restore a public testnet wallet
   i. if you don't have a testnet wallet, generate a recovery phrase with cabal exec -- cardano-wallet recovery-phrase generate
   ii. run curl -H "content-type: application/json" -XPOST -d @restore-wallet.json localhost:8090/v2/wallets where restore-wallet.json is a request body JSON file with the recovery phrase (example). Note the spending password!
   iii. If this is a new wallet, you need to use the testnet faucet to send some tAda to it. An address of the wallet can be obtained using the wallet's HTTP API. Verify that the wallet has received the tAda by calling getWallet
5. Start the chain index. cabal exec -- plutus-chain-index --config ${chain-index-config} start-index in a new terminal. The configuration should look something like this - don't forget to replace the ${plutus-apps} variable.
6. Start the PAB. ${pab-config}. cabal exec -- plutus-pab-examples --config ${pab-config} webserver --passphrase PABPABPABPAB (passphrase is the spending password from step 4.ii) in a new terminal
7. Wait for all chain clients (cardano-node, cardano-wallet, plutus-pab-examples, plutus-chain-index) to fully synchronise with the testnet. The chain index is probably going to take the longest time to synchronise, ~3h on my machine. To check the sync status of the chain index, run curl localhost:9083/tip | jq . When it's fully synched the slot should be the same slot that the plutus-pab-examples process last printed to the console. Note that the PAB process will be silent for a while, after the initial log messages, before it starts printing log messages about its sync status.
8. START THE INTEGRATION TEST curl -H "Content-Type: application/json" -v -X POST -d '{"caID":{"tag":"IntegrationTest"},"caWallet":{"getWalletId":"4c04ce356e087d12980a1e5134c15234661acc39"}}' localhost:9080/api/contract/activate (note the wallet ID - this has to be the ID of the wallet that you created in step 4)
9. Wait for the integration test to finish. There will be a message saying "Integration test complete" from the PAB process when it's done. You can use the Cardano explorer to check the transactions that were produced.

Things that can go wrong

• If you start the integration test before the chain index is fully synchronised, it will fail silently after the first transaction.
• If the chain index host & port aren't configured correctly in the PAB, the test will also fail silently after the first transaction.
• Possibly other things too (the error handling is not great)

Notes

• The node configuration used by the local cluster has much shorter epoch / slot times than the public testnet or mainnet
• It also seems to have more frequent rollbacks than those nets (which is good for us because this makes it easier to test the code in the presence of rollbacks)

TODO / Plans:

• Figure out a better way to get the test data from cardano-wallet --> add it to this PR
• Start up chain index
• Start up PAB with correct wallet config
• Add a test that creates and spends a script output
• PR to cardano-wallet
• Reduce log levels for wallet client

Pre-submit checklist:

• Branch
  • Tests are provided (if possible)
  • Commit sequence broadly makes sense
  • Key commits have useful messages
  • Relevant tickets are mentioned in commit messages
  • Formatting, materialized Nix files, PNG optimization, etc. are updated
• PR
  • Self-reviewed the diff
  • Useful pull request description
  • Reviewer requested

[catch-21]

PR description says to "set the SHELLEY_TEST_DATA environment variable to lib/shelley/test/data/cardano-node-shelley", but when I run cabal exec plutus-pab-local-cluster it says Please make sure the SHELLEY_TEST_DATA environment variable is set to 'plutus-pab/local-cluster/cluster-data/cardano-node-shelley' in the plutus-apps repository..
Would be good to be clarify we're using absolute paths for this environment variable.

[sjoerdvisscher]

This works so smoothly! I do see a lot of "Rolling back to ChainPoint" messages. You don't have to wait for these, but they can drown out the "Tx confirmed" message.

[j-mueller]

Rolling back to ChainPoint messages

The problem is that the chain index prints all log messages to stdout, it doesn't filter by log level. I added an issue in Jira to fix that

[j-mueller]

@james-iohk Updated the PR description

[silky]

Works for me too; awesome @j-mueller !

[catch-21]

@james-iohk Updated the PR description

Thanks @j-mueller. I think it'd be worth adding these details to the plutus-pab/README.md. Although, it's already quite large so maybe just mention the local-cluster there and add details in a new plutus-pab/local-cluster/README.md.

[iaverypadberg]

When running the cabal build plutus-pab-local-cluster cardano-node cardano-wallet command I had problems building the lzma-0.0.0.3 (lib) dependency. The error was that it could not find the lzma.h file. This file is apart of the XZ package. To fix, download the XZ package, and use the ./configure -> make -> make install commands. I hope this helps someone out!