Planning: parse out Orchard spec updates, halo2/orchard crates, and zcashd updates into Zebra work items

[dconnolly]

We need to digest the current:

• Zcash (Orchard) spec draft
• the halo2/orchard crate books (documentation)
• the halo2/orchard crate code
• ZIP 224 - Orchard Shielded Protocol: [draft] NU5 specification zcash/zips#441
• ZIP 225 - Version 5 Transaction Format: [proposed] [ZIP 225] Version 5 Transaction Format zcash/zips#440
• the relevant changes to zcashd

to suss out what tasks we need to do in Zebra to have it validate Orchard compatibly, as the spec evolves.

To start with, we need to go through the spec and suss out the high level things to implement, and then if they don't have a straightforward equivalent in Zebra to model off of, designate the 'needs design' parts. From there, we need to see if there any glaring 'unknowns' that need to be fleshed out or answered in order to decide how to design them or how to prioritize them in project-planning order.

Result:

a zenhub epic (#1598) populated with issues and a vague dependency graph of work to support Orchard

• the enumerated Orchard parts of the draft spec
• checklist of ZIPs in all the NUs
• bucket of tasks that Zebra needs to do, not reliant on the spec or ZIPs, to gracefully handle the network upgrade

[yaahc]

I've done an initial call with @str4d and @daira to discuss the Orchard API

We have a shared doc for notes from the meeting and future meetings (just a few action items for now): https://hackmd.io/@2aetCAHHS5yPrDQIgqGo2Q/B1ogq6MMu

Overall the API looks good. Care has clearly been taken to make sure the API is consistent with our sapling module in zebra-chain and we can generally think of the orchard crate as equivalent to those sapling/sprout modules within zebra-chain. The AuthorizedBundle (name subject to change) type is equivalent to ShieldedData.

The main concerns that we ran into revolve around batching and constructing transactions. We don't currently construct transactions anywhere within zebra, so it is harder for them to predict what constraints we might have, but we're mostly confident that a builder api + trait based hooks solution for constructing each individual transaction format should work best with both codebases.

For batching they said we will initially not be able to batch proofs from separate transactions, I think largely because of the recursive proving, but that this isn't a fundamental hurdle and so we should be able to support fully batched proving down the line. For now though the protocol itself batches proofs within each transaction by design, so we will hopefully not have any performance issues even initially without the optimal batching implementation.

[teor2345]

For batching they said we will initially not be able to batch proofs from separate transactions, I think largely because of the recursive proving, but that this isn't a fundamental hurdle and so we should be able to support fully batched proving down the line. For now though the protocol itself batches proofs within each transaction by design, so we will hopefully not have any performance issues even initially without the optimal batching implementation.

For performance testing, we should construct blocks with:

• lots of transactions
• a transaction with lots of inputs
• a transaction with lots of outputs
• a transaction with lots of inputs and outputs

We should be able to share this work and the test data with the zcashd team.

Note that mainnet actually contains some of these kinds of transactions, due to miner and funding stream outputs, transfers, and collection.

[teor2345]

It might also be helpful to pass data through the batcher with a batch size of 1, to make sure that the API is compatible with batching. (For example, the changes we needed to make to the script(?) API to support efficient caching of transaction data during batching.)

We probably won't implement that design until later, but it might be helpful to change the API now.

[teor2345]

I updated the links in the ticked description to the latest versions of everything

[dconnolly]

OK from looking at the spec updates, it looks like a lot of the work we have to do for Orchard is

• wrap halo2 crate call in a new Halo2Verifier async service, like we do for Groth16, no fancy batch math
• New Txv5
• Keep track of Orchard note commitment tree anchors and nullifiers in state
• A few new things to check in the existing TransactionVerifier (like new binding signature checks from value commitments with the new curves / RedDSA/ RedPallas, mostly implemented in orchard crate, which we’ll help a bit with to make code nice
  • RedPallas async verifier?
• Maybe the tweaks to TxID generation and stuff

[mpguerra]

Result: a zenhub epic populated with issues and a vague dependency graph of work to support Orchard

FWIW, the Epic was already created here: #1598 we just need to keep adding issues to it :)

[mpguerra]

I'll finish this one off and close it in the next few days. This involves creating and filling in the following epics:

• Orchard/NU5 changes in the spec - NU5 Tracking: The Spec #1856
• ZIPs active in Canopy or NU5, which Zebra hasn't implemented yet - NU5 Tracking: Required ZIPs #1854
• Zebra-specific tasks - NU5 Tracking: Zebra tasks #1853
• Pre-NU5 spec sections still active in Canopy or NU5, which Zebra hasn't implemented yet - NU5 Tracking: Pre-NU5 Spec work #1872

[mpguerra]

* [ ]  ZIPs active in Canopy or NU5, which Zebra hasn't implemented yet - #1854

This one is almost done, pending an answer on whether we need to create an issue for ZIP-209

[mpguerra]

I've finished going through the NU5 spec and adding items to #1856. This issue will need a second and third pair of eyes to help me identify potentially missing issues and mistakes.

[mpguerra]

Going to close this for now as I think this is a good first pass.