v1.0.0 release (#189)

* fix power gate (#128) * Recursive Verifier (#127) * tmp * transcript refactor * transcript refactor * tmp * tm[ * v1 * rm proof txt file * clippy * minor * fmt * tmp fix * fix * fix errors in merge * minor * file location update * clean up * clean up * fix grinding * lock/unlock proof in transcript * testing script * update script * hash test in rust * minor * script update * update script * update script * tmp * fix expander-exec * fmt * Lagrange Form Bi kzg update (#130) * tmp * clean up * fix clippy * fix clippy * add a bit more tests * minor --------- Co-authored-by: zhenfei <zhenfei.zhang@hotmail.com> * [impl] adding sumcheck-cuda (#131) * stats code & release binary (#135) * stats code * release binary * fix-binary-release (#136) * Reuse eq_evals in prover, verifier, and recursive verifier (#138) * reduce randomness and reuse eq evals * add recursion in ci * add recursion in ci * pcs trait v1 (#140) * pcs trait v1 * &mut for pcs scheme * remove &mut self for verify * add cache for CI * Update nightly_e2e.yml * Features: Orion implementation related peripheral building blocks (#142) * Feature: Merkle Tree implementation as Orion pcs building block (#143) * init commit for merkle tree implementation * benchmarked and optimized, shit runs fast * allocate rather than concat * use move for intermediate results that can be dropped later * shorten bench time * additional benchmarks * Zf/config refactor (#146) * config refactor wip * config refactor wip * refactor config wip * refactor config wip * macros done * refactor done * fmt * clippy auto fix * fix grinding feature * fix verifier scratch pad * PCS Trait V2 & Integration (#150) Second version of PCS trait and PCS for Expander GKR interface, integrating Raw PCS to linear GKR prover * Minor: Merging `FieldSerde` into `Field`, and deprecating `try_deserialize_from_ecc_format` (#151) Co-authored-by: Zhiyong Fang <zhiyong.fang.1997@gmail.com> * Minor: Typo fixing for GF2 and tree bench shorten (#152) * Minor: MISC changes including rust fmt, CI caches, and MT serde (#154) * init commit for extension field from/to-limbs methods (#159) * Minor: Traits for multilinear polynomials, PCS minor changes (#158) * reference (read-only/write) multilinear polynomial prototype * full set of testing for both ref and mut_ref mle * minor, prototype (mut)-multilinear-extension traits for ref-mle-polys * continue with (mutable)multilinear-extension trait implementation * pcs interface change to box dyn multilinear-extension, collateral changes included * minor, add one more trait method of ref to hypercube basis * minor, use impl to get around lifetime specification in place * pcs crs from deterministic seed in testing across mpi (#162) * Minor: Fixing BN254 Recursion on RawPCS deserialization (#163) * Expose expander-exec APIs for ECC (#165) * expose exec apis * fix clippy * fix clippy * Minor: Poseidon M31 Transcript (#160) Signed-off-by: Tiancheng Xie <tc@polyhedra.network> Co-authored-by: Tiancheng Xie <tc@polyhedra.network> * Zf/virgo++ (#164) * e2e infrastructure * tmp * tmp * circuit deserialization * tmp * tmp * v1 * fmt * clippy auto fix * clippy * minor * minor * fmt * minor * fix mpi ci (#174) * remove rust cache * try... * clean up * profiling * the old fashion * Update install.py * remove apt update * Update benchmarks.yml * enable sudo for CI * [impl] update soft-lock based naive prepare impl (#173) * temporarily uses GitHub's CI (#175) * Minor: Recursion Script Typo Fix from CI Fix (#176) * Minor: Orion PCS only base field impls (#177) * Minor: Mersenne31 GKR recursive verifier (#168) * extracted changes for poseidon m31 transcript * minor - mimc tested, sponge set/get state trait methods * fkin hell - just wanna specify Transcript baseF challengeF ... didnt expect this comin * morphing field hash transcript, hash from base field output challenge field * squeeze, squeeze_state, and set_state in field hash sponge trait * changed interface for transcript - wondering why recursive mimc transcript not matching * so poseidon m31 works for gkr correctness, which is a relief * you squeaky lil bug there you go * Revert "you squeaky lil bug there you go" This reverts commit 1bc8381. * Revert "so poseidon m31 works for gkr correctness, which is a relief" This reverts commit 573b7d4. * Revert "changed interface for transcript - wondering why recursive mimc transcript not matching" This reverts commit a8ff4f2. * Revert "squeeze, squeeze_state, and set_state in field hash sponge trait" This reverts commit ce80aec. * Revert "morphing field hash transcript, hash from base field output challenge field" This reverts commit 90ebc09. * Revert "fkin hell - just wanna specify Transcript baseF challengeF ... didnt expect this comin" This reverts commit 559b565. * revert back to field hasher from previous interface * minor rewrite on field hash transcript - move FiatShamirFieldHash into arith * minor changes stashing * refactoring field hash transcript to using hash state (vec of base field elems) * minor rewrite - make transcript impl less lines * minor - move mimc to crate field_hashers under arith, remove hasher mimc impl * prototyped poseidon permutation and integrated into gkr correctness * export with name suffix fiat-shamir-hasher, rename to align names * minor - mpi sync up state * minor - typo fixing * stashing changes for sampling circuit elems * gathering stashed circuit changes * recursion CI setup * stashing CI related changes and notes on commenting out api println * minor - api for zeroes over extension field * stash changes from transcript side * raw commitment length w.r.t. base field needs input size * mpi size * simd size * take simd variables into consideration, and note places needs fixing * continue code read with gkr runtime and check against recursive verifier * minor - setup circuit for recursive test * update go mod version * prototyped recursive verifier with CI running * minor - follow up changes for merging dev changes * CI recursion steps update * minor - upload a small enough circuit for recursion test * update ecc branch - remove prev poseidon impl * i dont like it but it works for CI for now * clean up ci yml file --------- Signed-off-by: Tiancheng Xie <tc@polyhedra.network> Co-authored-by: Tiancheng Xie <tc@polyhedra.network> * Minor: Fix CI build (#178) * Minor: SIMD orion PCS impls (#179) * Feature: Orion PCS Integration (#153) --------- Signed-off-by: Zhiyong Fang <zhiyong.fang.1997@gmail.com> Co-authored-by: Zhiyong Fang <zhiyong.fang.1997@gmail.com> * CI: build ECC locally rather than downloading (#181) * Minor: Simpler CI from ECC script (#182) * init commit for the gf2 avx256 fix (#183) * fix fiat-shamir (#184) * hash the commitment for sufficient depth * clean up * clean up * clean up * use recursion gate to guard the grinding * Minor: Unified CI workflows (#185) * Minor: Bringing benchmark machine back (#186) * Minor: PCS trait changes (#188) * Minor: Fixing build workflow CI process (#187) * Update readme.md Signed-off-by: Tiancheng Xie <tc@polyhedra.network> * Update readme.md Signed-off-by: Tiancheng Xie <tc@polyhedra.network> * fixes broken link (#147) --------- Signed-off-by: Tiancheng Xie <tc@polyhedra.network> Signed-off-by: Zhiyong Fang <zhiyong.fang.1997@gmail.com> Co-authored-by: Zhiyong Fang <zhiyong.fang.1997@gmail.com> Co-authored-by: Simon Lau <slzk@polyhedra.network> Co-authored-by: tonyfloatersu <tonyfloater@gmail.com> Co-authored-by: mcfx <git1@mcfx.us> Co-authored-by: Tiancheng Xie <tc@polyhedra.network> Co-authored-by: Yahya Hassanzadeh, Ph.D. <19204398+yhassanzadeh13@users.noreply.github.com>
PolyhedraZK · Jan 27, 2025 · d40fd2d · d40fd2d
1 parent bf6f78f
commit d40fd2d
Show file tree

Hide file tree

Showing 201 changed files with 12,884 additions and 4,205 deletions.
diff --git a/.github/workflows/benchmarks.yml b/.github/workflows/benchmarks.yml
@@ -1,8 +1,11 @@
 name: Run Benchmarks
 
-on:
-  push:
-  pull_request:
+env:
+  RUSTFLAGS: "-Dwarnings -C target-cpu=native"
+  RUST_BACKTRACE: 1
+  ACTIONS_RUNNER_DEBUG: true
+
+on: [push]
 
 jobs:
   benchmark:
@@ -11,8 +14,6 @@ jobs:
     - uses: actions/checkout@v4
       with:
         fetch-depth: 0
-    - name: Set up Python
-      run: python3 --version
     - name: update toolchain
       uses: actions-rs/toolchain@v1
       with:
@@ -27,15 +28,13 @@ jobs:
         project_id: ${{ secrets.GCP_PROJECT_ID }}
     - name: Install criterion
       run: cargo install cargo-criterion
-    - name: Install mpi
-      run: python3 ./scripts/install.py
-    - name: Setup
+    - name: Setup data
       run: cargo run --bin=dev-setup --release
     - name: Run benchmark
       run: RUSTFLAGS="-C target-cpu=native" cargo criterion --message-format=json > benchmark_results.json
     - name: Clean benchmark results
       run: |
-        wget https://raw.githubusercontent.com/PolyhedraZK/Expander-rs/main/scripts/parse_benchmark_result.py
+        wget https://raw.githubusercontent.com/PolyhedraZK/Expander/main/scripts/parse_benchmark_result.py
         python3 parse_benchmark_result.py benchmark_results.json benchmark_results_clean.json
     - name: Upload benchmark results to Google Cloud Storage
       run: |

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -6,6 +6,7 @@ on:
 env:
   RUSTFLAGS: "-Dwarnings -C target-cpu=native"
   RUST_BACKTRACE: 1
+  ACTIONS_RUNNER_DEBUG: true
 
 jobs:
   build:
@@ -20,21 +21,32 @@ jobs:
           - os: 7950x3d
             feature: avx512f
             binary_name: expander-linux-avx512
+
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
+
+      - name: Setup Dependencies for MacOS workflow
+        if: matrix.os == 'macos-latest'
+        run: brew install gcc make openmpi
+
       - uses: dtolnay/rust-toolchain@stable
+
       - uses: Swatinem/rust-cache@v2
         with:
-          # The prefix cache key, this can be changed to start a new cache manually.
-          prefix-key: "mpi-v5.0.5" # update me if brew formula changes to a new version
-      - name: Install MPI
+          prefix-key: "mpi-v5.0.6"
+
+      - name: Install MPI for MacOS workflow
+        if: matrix.os == 'macos-latest'
         run: python3 ./scripts/install.py
+
       - name: Set RUSTFLAGS for AVX
-        if: matrix.feature != ''
-        run: echo "RUSTFLAGS=$RUSTFLAGS -C target-feature=+${{ matrix.feature }}" >> $GITHUB_ENV  
+        if: matrix.feature != 'macos-latest'
+        run: echo "RUSTFLAGS=$RUSTFLAGS -C target-feature=+${{ matrix.feature }}" >> $GITHUB_ENV
+
       - name: Prepare binary
         run: cargo build --release --bin expander-exec
+
       - name: Upload release asset
         uses: actions/github-script@v6
         with:

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -1,96 +1,105 @@
+# Expander CI: unit tests on Github's servers
+# it runs on both macos and ubuntu avx2
+# it does not check for avx512 -- this is not available on github
+
 name: expander-rs CI
 
 on: [pull_request, push]
 
 env:
   RUSTFLAGS: "-Dwarnings -C target-cpu=native"
+  CARGO_TERM_COLOR: always
   RUST_BACKTRACE: 1
+  ACTIONS_RUNNER_DEBUG: true
 
 jobs:
   lint:
     name: Lint
-    runs-on: 7950x3d
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@stable
         with:
           components: rustfmt, clippy
-      - run: python3 ./scripts/install.py
-      - run: cargo fmt --all -- --check
-      - run: cargo clippy --all
+
+      - name: Setup Dependencies
+        run: sudo apt-get install -y build-essential openmpi-bin libopenmpi-dev
+
+      - name: Check formatting
+        run: cargo fmt --all -- --check
+
+      - name: Run clippy
+        run: cargo clippy --all
 
   build-and-test:
-    name: Build and Test (${{ matrix.os }}${{ matrix.feature != '' && format(', {0}', matrix.feature) || '' }})
+    name: Build and Test (${{ matrix.os }})
     runs-on: ${{ matrix.os }}
     strategy:
+      fail-fast: false
       matrix:
         include:
           - os: macos-latest
-          - os: 7950x3d
-            feature: avx2
+          - os: ubuntu-latest
           - os: 7950x3d
             feature: avx512f
+
     steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
-      - uses: Swatinem/rust-cache@v2
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Cache Rust dependencies
+        uses: Swatinem/rust-cache@v2
         with:
-          # The prefix cache key, this can be changed to start a new cache manually.
-          prefix-key: "mpi-v5.0.5" # update me if brew formula changes to a new version
-      - name: Set RUSTFLAGS for AVX
-        if: matrix.feature != ''
-        run: echo "RUSTFLAGS=$RUSTFLAGS -C target-feature=+${{ matrix.feature }}" >> $GITHUB_ENV  
-      - name: Build and Test
-        run: |
-          python3 ./scripts/install.py
-          mpiexec --version
-          cargo run --bin=dev-setup --release
-          cargo build --all-features --release
-          cargo test --all-features --release --workspace
-          ./scripts/test_recursion.py
-
-  gkr-e2e:
-    name: Benchmark (${{ matrix.os }}${{ matrix.feature != '' && format(', {0}', matrix.feature) || '' }}, ${{ matrix.field }})
-    runs-on: ${{ matrix.os }}
-    strategy:
-      matrix:
-        include:
-          - os: macos-latest
-            field: m31ext3
-          - os: macos-latest
-            field: fr
-          - os: macos-latest
-            field: gf2ext128
-          - os: 7950x3d
-            feature: avx2
-            field: m31ext3
-          - os: 7950x3d
-            feature: avx2
-            field: fr
-          - os: 7950x3d
-            feature: avx2
-            field: gf2ext128
-          - os: 7950x3d
-            feature: avx512f
-            field: m31ext3
-          - os: 7950x3d
-            feature: avx512f
-            field: fr
-          - os: 7950x3d
-            feature: avx512f
-            field: gf2ext128
-    steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
-      - uses: Swatinem/rust-cache@v2
+          prefix-key: "mpi-v5.0.6"
+
+      - uses: actions/setup-go@v5
+        if: matrix.os != '7950x3d'
         with:
-          # The prefix cache key, this can be changed to start a new cache manually.
-          prefix-key: "mpi-v5.0.5" # update me if brew formula changes to a new version
-      - name: Set RUSTFLAGS for AVX
-        if: matrix.feature != ''
-        run: echo "RUSTFLAGS=$RUSTFLAGS -C target-feature=+${{ matrix.feature }}" >> $GITHUB_ENV
-      - name: Run benchmark
+          go-version: '>=1.22.0'
+
+      - name: Setup Dependencies for MacOS workflow
+        if: matrix.os == 'macos-latest'
+        run: brew install gcc make openmpi
+
+      - name: Setup Dependencies for Ubuntu workflow
+        if: matrix.os == 'ubuntu-latest'
+        run: sudo apt-get install -y build-essential openmpi-bin libopenmpi-dev
+
+      - name: Setup ECC Library
+        run: |
+          cd ~
+          rm -rf ExpanderCompilerCollection
+          git clone -b dev https://github.com/PolyhedraZK/ExpanderCompilerCollection.git
+          cd ExpanderCompilerCollection
+          ./build-rust.sh
+
+      - name: Download test data
+        run: cargo run --bin=dev-setup --release
+
+      - name: Build project
+        run: cargo build --all-features --release
+
+      - name: Run unit tests
+        run: cargo test --all-features --release --workspace
+
+      - name: Run E2E tests
+        run: ./scripts/test_recursion.py
+
+      - name: Run benchmarks on Github Workflows
+        if: matrix.os != '7950x3d'
+        run: |
+          cargo run --bin=gkr --release -- -t 4 -f gf2ext128
+          cargo run --bin=gkr --release -- -t 4 -f m31ext3
+          cargo run --bin=gkr --release -- -t 4 -f fr
+
+      - name: Run benchmarks on AVX512 7950x3d
+        if: matrix.os == '7950x3d'
         run: |
-          python3 ./scripts/install.py
-          cargo run --bin=dev-setup --release
-          cargo run --bin=gkr --release -- -t ${{ matrix.os == 'macos-latest' && 2 || 16 }} -f ${{ matrix.field }}
+          cargo run --bin=gkr --release -- -t 16 -f gf2ext128
+          cargo run --bin=gkr --release -- -t 16 -f m31ext3
+          cargo run --bin=gkr --release -- -t 16 -f fr
diff --git a/.github/workflows/nightly_e2e.yml → .github/workflows/nightly_e2e._yml b/.github/workflows/nightly_e2e.yml → .github/workflows/nightly_e2e._yml
@@ -1,10 +1,15 @@
 name: E2E Tests
 
+env:
+  RUSTFLAGS: "-Dwarnings -C target-cpu=native"
+  RUST_BACKTRACE: 1
+  ACTIONS_RUNNER_DEBUG: true
+
 on:
   schedule:
     # This will run the workflow every day at 2:00 AM UTC
     - cron: '0 2 * * *'
-  
+
   pull_request:
     branches:
     # not to be run on DEV branch to save resources
@@ -16,24 +21,20 @@ on:
 jobs:
   build-and-test:
     runs-on: 7950x3d
-    
+
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4  # Updated to v4
 
     - name: Set up Rust
       uses: dtolnay/rust-toolchain@master
       with:
         toolchain: nightly-2024-09-01
         components: rustfmt, clippy
+
     - uses: Swatinem/rust-cache@v2
       with:
         # The prefix cache key, this can be changed to start a new cache manually.
-        prefix-key: "mpi-v5.0.5" # update me if brew formula changes to a new version
-
-    - name: Set up Go
-      uses: actions/setup-go@v2
-      with:
-        go-version: '^1.16'
+        prefix-key: "mpi-v5.0.6" # update me if brew formula changes to a new version
 
     - name: Run tests
       run: |
@@ -42,7 +43,7 @@ jobs:
 
         # ECC part
         git clone https://github.com/PolyhedraZK/ExpanderCompilerCollection.git
-        cd ExpanderCompilerCollection 
+        cd ExpanderCompilerCollection
         git switch dev
         cargo test --release
         cp expander_compiler/*.txt ../data
@@ -66,9 +67,3 @@ jobs:
         RUSTFLAGS="-C target-cpu=native -C target-feature=+avx512f" cargo +nightly run --release --bin=gkr -- -s keccak -f m31ext3 -t 16
         RUSTFLAGS="-C target-cpu=native -C target-feature=+avx512f" cargo +nightly run --release --bin=gkr -- -s keccak -f fr -t 16
         RUSTFLAGS="-C target-cpu=native -C target-feature=+avx512f" cargo +nightly run --release --bin=gkr -- -s poseidon -f m31ext3 -t 16
-
-        # # Run MPI tests
-        # RUSTFLAGS="-C target-cpu=native -C target-feature=+avx512f" mpiexec -n 2 cargo +nightly run --release --bin=gkr-mpi -- -s keccak -f gf2ext128
-        # RUSTFLAGS="-C target-cpu=native -C target-feature=+avx512f" mpiexec -n 2 cargo +nightly run --release --bin=gkr-mpi -- -s keccak -f m31ext3
-        # RUSTFLAGS="-C target-cpu=native -C target-feature=+avx512f" mpiexec -n 2 cargo +nightly run --release --bin=gkr-mpi -- -s keccak -f fr
-        # RUSTFLAGS="-C target-cpu=native -C target-feature=+avx512f" mpiexec -n 2 cargo +nightly run --release --bin=gkr-mpi -- -s poseidon -f m31ext3