From 1ae1bbd203824513995ff5a745d495074629a72b Mon Sep 17 00:00:00 2001 From: Nick Date: Thu, 18 May 2023 20:39:46 +0300 Subject: [PATCH] docs: update readme with instructions for Dockerized environment Update two sections: * Introduction: cloning the repository and getting Docker images. * Local development workflow: compile a circuit, make a statement, make a proof. Add a table of contents, made with `doctoc`: ```bash npm install -g doctoc doctoc README.md ``` The section with Proof Market is almost untouched and very much out of date. Will update it in #21 Resolve #4 Resolve #18i --- README.md | 310 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 261 insertions(+), 49 deletions(-) diff --git a/README.md b/README.md index a942195..10b1a9e 100644 --- a/README.md +++ b/README.md @@ -1,82 +1,294 @@ -# zkLLVM Template + +[![Tutorial check](https://github.com/NilFoundation/zkllvm-template/actions/workflows/main.yml/badge.svg)](https://github.com/NilFoundation/zkllvm-template/actions/workflows/main.yml) + +# zkLLVM Tutorial and template project Tutorial and a template repository for a zk-enabled application project based on the [zkLLVM toolchain](https://github.com/nilfoundation/zkllvm). - Use it to learn about developing zk-enabled apps with zkLLVM step-by-step. -Code in `./src` is an example of BLS12-381 signature verification via zkLLVM using -[Crypto3 C++ cryptography suite](https://github.com/nilfoundation/crypto3) as an SDK. +For this tutorial you will need an amd64 machine with Docker (on Linux) or Docker Desktop (on macOS). + -# Preparing environment with zkLLVM + + +**Table of Contents** + +- [Preparing environment for the tutorial](#preparing-environment-for-the-tutorial) + - [1. Clone the template repository and submodules](#1-clone-the-template-repository-and-submodules) + - [2. Get the Docker images with =nil; toolchain](#2-get-the-docker-images-with-nil-toolchain) +- [Part 1. Local development workflow](#part-1-local-development-workflow) + - [Step 1: Compile a circuit](#step-1-compile-a-circuit) + - [Step 2: Build a circuit statement](#step-2-build-a-circuit-statement) + - [Step 3: Produce and verify a proof locally](#step-3-produce-and-verify-a-proof-locally) +- [Part 2. Proof Market workflow](#part-2-proof-market-workflow) + - [Step 2: Setup proof market user/toolchain](#step-2-setup-proof-market-usertoolchain) + - [Step 2: Prepare circuit to publish to Proof Market](#step-2-prepare-circuit-to-publish-to-proof-market) + - [Step 4: See All Published circuits](#step-4-see-all-published-circuits) + - [Step 5: Push a Bid](#step-5-push-a-bid) + - [Step 6: Push an Ask](#step-6-push-an-ask) + - [Step 7 : Fetch statements/inputs for proof generation](#step-7--fetch-statementsinputs-for-proof-generation) + - [Step 8 : Generate Proof](#step-8--generate-proof) + - [Step 9: Publish Proof](#step-9-publish-proof) +- [Common issues](#common-issues) + - [Compilation errors](#compilation-errors) + + + + +# Preparing environment for the tutorial + +There are a few steps ## 1. Clone the template repository and submodules First, clone this repository with all its submodules: -``` +```bash git clone --recurse-submodules git@github.com:NilFoundation/zkllvm-template.git cd zkllvm-template ``` -## 2. Run a Docker container +If you cloned without `--recurse-submodules`, initialize submodules explicitly: -For tutorial purposes, we will do everything in Docker. +```bash +git submodule update --init --recursive +``` -First, run a new container named `zkllvm` -and mount this project's directory in it: +## 2. Get the Docker images with =nil; toolchain -```console -$ docker run --name zkllvm \ - --platform=linux/amd64 -it \ - -v $(pwd):/opt/zkllvm-template \ - ubuntu:latest - -# cd /opt/zkllvm-template -``` +We've prepared Docker images with parts of the `=nil;` toolchain +that you'll need to run through all development stages with this template. +We recommend using them, because they're tested for compatibility, +and they save you time on installing and compiling everything. -## 3. Install zkLLVM and dependencies +* `nilfoundation/zkllvm-template` has the zkLLVM part of the toolchain: + the compiler (`clang`), `assigner`, and `tranpiler` binaries. +* `nilfoundation/proof-market-toolchain` has all you need to make an account on + the `=nil;` Proof Market, put your circuit on it, and order a proof. -zkLLVM is distributed as a deb package, but we need to setup the repository first: +These images are versioned according to the products they contain. +In the tutorial, we'll use the latest versions that are compatible with each other. +```bash +export ZKLLVM_VERSION=0.0.58 +export TOOLCHAIN_VERSION=0.0.31 +docker pull ghcr.io/nilfoundation/zkllvm-template:${ZKLLVM_VERSION} +docker pull ghcr.io/nilfoundation/proof-market-toolchain:${TOOLCHAIN_VERSION} ``` -echo 'deb [trusted=yes] http://deb.nil.foundation/ubuntu/ all main' >>/etc/apt/sources.list -apt update -apt install -y zkllvm cmake libboost-all-dev + +# Part 1. Local development workflow + +In the first part of this tutorial, we'll walk through the development workflow on +a local machine, without using Proof Market. +We will build a circuit, pack it into a circuit statement, +and then use it to verify a proof for particular input. + +You will run each step as a command, conveniently wrapped in the `./scripts/run.sh` script. +We recommend using it when you do the tutorial for the first time. + +Once you've completed the tutorial, you can repeat it by running all commands manually in the console. +Look at the `[manual mode]` instructions in the collapsed blocks. + +Code in `./src` is an example of BLS12-381 signature verification via zkLLVM using +[Crypto3 C++ cryptography suite](https://github.com/nilfoundation/crypto3) as an SDK. + +## Step 1: Compile a circuit + +In `./src/main.cpp`, we have a function starting with `[[circuit]]`. +This code definition is what we call the circuit itself. +We will use zkLLVM compiler to make a byte-code representation of this circuit. + +Run the script from the root of your project. + +```bash +./scripts/run.sh --docker compile ``` -Note that zkLLVM replaces original clang, being a fully compatible drop-in replacement: +The `compile` command does the following: + +1. Starts a Docker container based on `nilfoundation/zkllvm-template`. +2. Makes a clean `./build` directory and initializes `cmake`. +3. Compiles the code into a circuit. + +
+ [manual mode] + +Start a Docker container with zkLLVM toolchain: + ``` -# clang --version +docker run -it --rm \ + --platform=linux/amd64 \ + --volume $(pwd):/opt/zkllvm-template \ + --user $(id -u ${USER}):$(id -g ${USER}) \ + ghcr.io/nilfoundation/zkllvm-template:0.0.58 +``` +(Note that it's a single command, wrapped on several lines.) + +> The line `--volume $(pwd):/opt/zkllvm-template` mounts the project directory from +your host machine into the container, so that the source code is available in it. +All the changes will persist on your machine, +so you can stop this container at any time, start a new one, and continue. + +> The line `--user $(id -u ${USER}):$(id -g ${USER})` runs container with a user having +the same user and group ID as your own user on the host machine. +With this option, newly created files in the `./build` directory will +belong to your user, and not to the `root`. + +Let's check that we have the zkLLVM compiler available in the container. +Note that it replaces original `clang`, being a fully compatible drop-in replacement: + +```console +$ clang --version clang version 16.0.0 (https://github.com/NilFoundation/zkllvm-circifier.git bf352a2e14522504a0c832f2b66f73268c95e621) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /usr/bin ``` -## 4. Stop and restart container when needed +In the container, create a `./build` directory and compile the code: -This new Docker container persists when you exit it. -Restart it when you need it again, -and you won't need to reinstall zkLLVM and other dependencies every time. +```bash +cd /opt/zkllvm-template +rm -rf build && mkdir build && cd build +cmake -DCIRCUIT_ASSEMBLY_OUTPUT=TRUE .. +make template +``` -```console -$ docker container start -i zkllvm - -root@9ef17682eaca:/# cd /opt/zkllvm-template +> The extra parameter `DCIRCUIT_ASSEMBLY_OUTPUT=TRUE` is required to produce circuits +in `.ll` format, which is supported by proving tools. +zkLLVM can also produce circuits in another LLVM's IR format, `.bc`, but we won't need it in this tutorial. + +
+ +As a result of this step, we get a byte-code file `./build/src/template.ll`. +This is what we call a circuit itself. +It's a binary file in the LLVM's intermediate representation format. + +## Step 2: Build a circuit statement + +Proof Market works with circuits in the form of circuit statements. +A statement is basically a JSON with the circuit and various metadata that identifies it. + +The `build_statement` command will build a circuit statement from the circuit that we compiled earlier: + +```bash +./scripts/run.sh --docker build_statement ``` -# Zero-knowledge proof workflow +The `build_statement` command does the following: -# Step 1 : Build the intermediate representation (IR) file -``` -mkdir build && cd build -cmake .. && make template +1. Runs a new container based on `nilfoundation/proof-market-toolchain`. +2. In the container, runs `prepare_statement.py` to produce a circuit statement. + +
+ [manual mode] + +To build a statement, we will use the `prepare_statement.py` script, +which is a part of the [Proof Market toolchain](https://github.com/nilfoundation/proof-market-toolchain). + +First, start a new container with the Proof Market toolchain. +(Remember to exit the `zkllvm` container with `exit` command or start a new console session): + +```bash +docker run -it --rm \ + --platform=linux/amd64 \ + --volume $(pwd):/opt/zkllvm-template \ + --volume $(pwd)/.config:/.config/ \ + --volume $(pwd)/.config:/root/.config/ \ + --user $(id -u ${USER}):$(id -g ${USER}) \ + ghcr.io/nilfoundation/proof-market-toolchain:0.0.31 +``` + +> The `.config` directory is where you will put the credentials to the Proof Market later on. +Two extra volume mounts make this directory available in places where +parts of the Proof Market toolchain might look for it. + +Now pack the circuit into a statement: + +```bash +cd /opt/zkllvm-template/ +python3 \ + /proof-market-toolchain/scripts/prepare_statement.py \ + -c /opt/zkllvm-template/build/src/template.ll \ + -o /opt/zkllvm-template/build/template.json \ + -n template \ + -t placeholder-zkllvm + +# -c: path to the circuit file +# -o: path to write the statement file +# -n: statement name +# -t: type of proofs that will be generated with this statement +# (Placeholder is the name of our proof system, see +# https://crypto3.nil.foundation/papers/placeholder.pdf) +``` +
+ +As a result, we have the circuit statement file `./build/template.json`. +Later we will use it to generate a proof locally, and then push the statement to the Proof Market. + +## Step 3: Produce and verify a proof locally + +Now we have everything ready to produce our first proof. +We'll use the `proof-generator` CLI, which is a part of the Proof Market toolchain. + +```bash +./scripts/run.sh --docker prove ``` -You should have a circuit IR file called `template.ll` -# Step 2: Setup proof market user/toolchain -Please navigate out of the `zkllvm-template` repository +
+ [manual mode] + +Continue in the `proof-market-toolchain` container that you made in step 2. + +First, create an empty configuration file in your user's directory. +Later you will put the Proof Market credentials in it. + +```bash +touch /opt/zkllvm-template/.config/config.ini +``` + +Next, run the `proof-generator` binary to generate a proof: + +```bash +proof-generator \ + --circuit_input=/opt/zkllvm-template/build/template.json \ + --public_input=/opt/zkllvm-template/src/main-input.json \ + --proof_out=/opt/zkllvm-template/build/template.proof + +# --circuit_input: path to the circuit statement +# --public_input: path to the file that contains particular input, that we want to make a proof for +# --proof_out: path and name of the proof file +``` +
+ +Note the following lines in the build log: + +``` +generatring zkllvm proof... +Proof is verified +``` + +In the first line `proof-generator` creates a proof, and in the second — verifies it. +The resulting proof is in the file `./build/template.proof`. + +Congratulations! +You've produced a non-interactive zero-knowledge proof, or, formally speaking, +a zero-knowledge succinct non-interactive argument of knowledge +([zk-SNARK](https://en.wikipedia.org/wiki/Non-interactive_zero-knowledge_proof)). + +Now it's time to work with the =nil; Proof Market. + + +# Part 2. Proof Market workflow + +In this part, we will interact with the Proof Market in two roles: + +* As a circuit developer, who puts a circuit on the market. +* As a zkApp developer, who orders proofs from the market. + +## Step 2: Setup proof market user/toolchain +Please navigate out of the `zkllvm-tfemplate` repository Clone the proof-market-toolchain repository @@ -105,33 +317,33 @@ username password ``` -# Step 3 :Prepare circuit to publish to Proof Market +## Step 2: Prepare circuit to publish to Proof Market ``` python3 scripts/prepare_statement.py -c=/root/tmp/zkllvm/build/examples/arithmetics_example.ll -o=arithmetic.json -n=arithmetic -t=placeholder-zkllvm ``` - -# Step 4: See All Published circuits + +## Step 4: See All Published circuits ``` python3 scripts/statement_tools.py get ``` -# Step 5: Push a Bid +## Step 5: Push a Bid ``` python3 scripts/bid_tools.py push --cost --file --key ``` -# Step 6: Push an Ask +## Step 6: Push an Ask ``` python3 scripts/ask_tools.py push --cost --key ``` -# Step 7 : Fetch statements/inputs for proof generation +## Step 7 : Fetch statements/inputs for proof generation ``` python3 scripts/statement_tools.py get --key -o python3 scripts/public_input_get.py --key -o ``` -# Step 8 : Generate Proof +## Step 8 : Generate Proof - Build proof generator ``` mkdir build && cd build @@ -144,7 +356,7 @@ cmake --build . -t proof-generator ./bin/proof-generator/proof-generator --proof_out=/root/workshop/arith_proof.out --circuit_input=/root/tmp/proof-market-toolchain/arithmetic.json --public_input=/root/tmp/proof-market-toolchain/example/input/arithmetic_example/input.json ``` -# Step 9: Publish Proof +## Step 9: Publish Proof ``` python3 scripts/proof_tools.py push --bid_key --ask_key --file ```