Add golang bindings for hash + merkle tree APIs

.github/workflows/golang.yml

@@ -76,7 +76,7 @@ jobs:
       if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
       # builds a single curve with the curve's specified build args
       run: |
-        ./build.sh -curve=${{ matrix.curve.name }} ${{ matrix.curve.build_args }} -cuda_backend=local
+        ./build.sh -curve=${{ matrix.curve.name }} ${{ matrix.curve.build_args }} -cuda_backend=local -skip_hash
     - name: Test
       working-directory: ./wrappers/golang/curves
       if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
@@ -114,11 +114,49 @@ jobs:
       if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
       # builds a single field with the fields specified build args
       run:  |
-        ./build.sh -field=${{ matrix.field.name }} ${{ matrix.field.build_args }} -cuda_backend=local
+        ./build.sh -field=${{ matrix.field.name }} ${{ matrix.field.build_args }} -cuda_backend=local -skip_hash
     - name: Test
       working-directory: ./wrappers/golang/fields
       if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
       run: |
         FIELD=$(echo ${{ matrix.field.name }} | sed -e 's/_//g')
         export ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib
         go test ./$FIELD/tests -count=1 -failfast -p 2 -timeout 60m -v
+
+  build-hash-linux:
+    name: Build and test hash on Linux
+    runs-on: [self-hosted, Linux, X64, icicle]
+    needs: [check-changed-files, check-format, extract-cuda-backend-branch]
+    steps:
+    - name: Checkout Repo
+      uses: actions/checkout@v4
+    - name: Checkout CUDA Backend
+      if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
+      uses: actions/checkout@v4
+      with:
+        repository: ingonyama-zk/icicle-cuda-backend
+        path: ./icicle/backend/cuda
+        ssh-key: ${{ secrets.CUDA_PULL_KEY }}
+        ref: ${{ needs.extract-cuda-backend-branch.outputs.cuda-backend-branch }}
+    - name: Setup go
+      uses: actions/setup-go@v5
+      with:
+        go-version: '1.22.0'
+    - name: Build
+      working-directory: ./wrappers/golang
+      if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
+      # builds the hash and merkle tree lib using a local copy of the CUDA backend
+      run:  |
+        ./build.sh -cuda_backend=local
+    - name: Test Hashes
+      working-directory: ./wrappers/golang/hash
+      if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
+      run: |
+        export ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib
+        go test ./tests -count=1 -failfast -p 2 -timeout 60m -v
+    - name: Test Merkle Tree
+      working-directory: ./wrappers/golang/merkle-tree
+      if: needs.check-changed-files.outputs.golang == 'true' || needs.check-changed-files.outputs.cpp == 'true'
+      run: |
+        export ICICLE_BACKEND_INSTALL_DIR=/usr/local/lib
+        go test ./tests -count=1 -failfast -p 2 -timeout 60m -v

docs/docs/icicle/golang-bindings/hash.md

@@ -0,0 +1,74 @@
+# ICICLE Hashing in Golang
+
+:::note
+
+For a general overview of ICICLE's hashing logic and supported algorithms, check out the [ICICLE Hashing Overview](../primitives/hash.md).
+
+:::
+
+:::caution Warning
+
+Using the Hash package requires `go` version 1.22
+
+:::
+
+## Overview
+
+The ICICLE library provides Golang bindings for hashing using a variety of cryptographic hash functions. These hash functions are optimized for both general-purpose data and cryptographic operations such as multi-scalar multiplication, commitment generation, and Merkle tree construction.
+
+This guide will show you how to use the ICICLE hashing API in Golang with examples for common hash algorithms, such as Keccak-256, Keccak-512, SHA3-256, SHA3-512, and Blake2s.
+
+## Importing Hash Functions
+
+To use the hashing functions in Golang, you only need to import the hash package from the ICICLE Golang bindings. For example:
+
+```go
+import (
+  "github.com/ingonyama-zk/icicle/v3/wrappers/golang/hash"
+)
+```
+
+## API Usage
+
+### 1. Creating a Hasher Instance
+
+Each hash algorithm can be instantiated by calling its respective constructor. The `New<Hash>Hasher` function takes an optional default input size, which can be set to 0 unless required for a specific use case.
+
+Example for Keccak-256:
+
+```go
+keccakHasher := hash.NewKeccak256Hasher(0 /* default input size */)
+```
+
+### 2. Hashing a Simple String
+
+Once you have created a hasher instance, you can hash any input data, such as strings or byte arrays, and store the result in an output buffer.
+Here’s how to hash a simple string using Keccak-256:
+
+```go
+import (
+  "encoding/hex"
+
+  "github.com/ingonyama-zk/icicle/v3/wrappers/golang/hash"
+)
+
+inputStrAsBytes := []bytes("I like ICICLE! It's so fast and simple")
+keccakHasher, error := hash.NewKeccak256Hasher(0 /*default chunk size*/)
+if error != runtime.Success {
+  fmt.Println("error:", error)
+  return
+}
+
+outputRef := make([]byte, 32)
+keccakHasher.Hash(
+  core.HostSliceFromElements(inputStrAsBytes),
+  core.HostSliceFromElements(outputRef),
+  core.GetDefaultHashConfig(),
+)
+
+// convert the output to a hex string for easy readability
+outputAsHexStr = hex.EncodeToString(outputRef)
+fmt.Println!("Hash(`", input_str, "`) =", &outputAsHexStr)
+```
+
+Using other hash algorithms is similar and only requires replacing the Hasher constructor with the relevant hashing algorithm.

docs/docs/icicle/golang-bindings/merkle.md

@@ -0,0 +1,242 @@
+# Merkle Tree API Documentation (Golang)
+
+This is the Golang version of the **Merkle Tree API Documentation** ([C++ documentation](../primitives/merkle.md)). It mirrors the structure and functionality of the C++ version, providing equivalent APIs in Golang.
+For more detailed explanations, refer to the [C++ documentation](../primitives/merkle.md).
+
+To see a complete implementation, visit the [Hash and Merkle example](https://github.com/ingonyama-zk/icicle/tree/main/examples/rust/hash-and-merkle) for a full example.
+
+:::caution Warning
+
+Using the Hash package requires `go` version 1.22
+
+:::
+
+## Tree Structure and Configuration in Golang
+
+### Defining a Merkle Tree
+
+```go
+/// * `layerHashers` - A vector of hash objects representing the hashers of each layer.
+/// * `leafElementSize` - Size of each leaf element.
+/// * `outputStoreMinLayer` - Minimum layer at which the output is stored.
+///
+/// # Returns a new `MerkleTree` instance or EIcicleError.
+func CreateMerkleTree(
+  layerHashers []hash.Hasher,
+  leafElementSize,
+  outputStoreMinLayer uint64,
+) (MerkleTree, runtime.EIcicleError)
+```
+
+The `outputStoreMinLayer` parameter defines the lowest layer that will be stored in memory. Layers below this value will not be stored, saving memory at the cost of additional computation when proofs are generated.
+
+### Building the Tree
+
+The Merkle tree can be constructed from input data of any type, allowing flexibility in its usage. The size of the input must align with the tree structure defined by the hash layers and leaf size. If the input size does not match the expected size, padding may be applied.
+
+Refer to the [Padding Section](#padding) for more details on how mismatched input sizes are handled.
+
+```go
+/// * `mt` - The merkle tree object to build
+/// * `leaves` - A slice of leaves (input data).
+/// * `config` - Configuration for the Merkle tree.
+///
+/// # Returns a result indicating success or failure.
+func BuildMerkleTree[T any](
+  mt *MerkleTree,
+  leaves core.HostOrDeviceSlice,
+  cfg core.MerkleTreeConfig,
+) runtime.EIcicleError
+```
+
+## Tree Examples in Golang
+
+### Example A: Binary Tree
+
+A binary tree with **5 layers**, using **Keccak-256**:
+
+![Merkle Tree Diagram](../primitives/merkle_diagrams/diagram1.png)
+
+```go
+import (
+  "github.com/ingonyama-zk/icicle/v3/wrappers/golang/core"
+  "github.com/ingonyama-zk/icicle/v3/wrappers/golang/hash"
+  merkletree "github.com/ingonyama-zk/icicle/v3/wrappers/golang/merkle-tree"
+)
+
+leafSize := 1024
+maxInputSize := leafSize * 16
+input := make([]byte, maxInputSize)
+
+hasher, _ := hash.NewKeccak256Hasher(uint64(leafSize))
+compress, _ := hash.NewKeccak256Hasher(2 * hasher.OutputSize())
+layerHashers := []hash.Hasher{hasher, compress, compress, compress, compress}
+
+mt, _ := merkletree.CreateMerkleTree(layerHashers, uint64(leafSize), 0 /* min layer to store */)
+
+merkletree.BuildMerkleTree[byte](&mt, core.HostSliceFromElements(input), core.GetDefaultMerkleTreeConfig())
+```
+
+### Example B: Tree with Arity 4
+
+![Merkle Tree Diagram](../primitives/merkle_diagrams/diagram2.png)
+
+This example uses **Blake2s** in upper layers:
+
+```go
+// define layer hashers
+// we want one hash layer to hash every 1KB to 32B then compress every 128B so only 2 more layers
+hasher, _ := hash.NewKeccak256Hasher(uint64(leafSize))
+compress, _ := hash.NewBlake2sHasher(2 * hasher.OutputSize())
+layerHashers := []hash.Hasher{hasher, compress, compress,}
+
+mt, _ := merkletree.CreateMerkleTree(layerHashers, uint64(leafSize), 0 /* min layer to store */)
+
+merkletree.BuildMerkleTree[byte](&mt, core.HostSliceFromElements(input), core.GetDefaultMerkleTreeConfig())
+```
+
+## Padding
+
+:::note
+Padding feature is not yet supported in **v3.1** and is planned for **v3.2**.
+:::
+
+When the input for **layer 0** is smaller than expected, ICICLE can apply **padding** to align the data.
+
+**Padding Schemes:**
+
+1. **Zero padding:** Adds zeroes to the remaining space.
+2. **Repeat last leaf:** The final leaf element is repeated to fill the remaining space.
+
+```go
+// type PaddingPolicy = int
+
+// const (
+//  NoPadding        PaddingPolicy = iota // No padding, assume input is correctly sized.
+//  ZeroPadding                           // Pad the input with zeroes to fit the expected input size.
+//  LastValuePadding                      // Pad the input by repeating the last value.
+// )
+
+import (
+  "github.com/ingonyama-zk/icicle/v3/wrappers/golang/core"
+)
+
+config := core.GetDefaultMerkleTreeConfig();
+config.PaddingPolicy = core.ZeroPadding;
+merkletree.BuildMerkleTree[byte](&mt, core.HostSliceFromElements(input), core.GetDefaultMerkleTreeConfig())
+```
+
+## Root as Commitment
+
+Retrieve the Merkle-root and serialize.
+
+```go
+/// Retrieve the root of the Merkle tree.
+///
+/// # Returns
+/// A reference to the root hash.
+func GetMerkleTreeRoot[T any](mt *MerkleTree) ([]T, runtime.EIcicleError)
+
+commitment := merkletree.GetMerkleTreeRoot[byte](&mt)
+fmt.Println!("Commitment:", commitment)
+```
+
+:::warning
+The commitment can be serialized to the proof. This is not handled by ICICLE.
+:::
+
+## Generating Merkle Proofs
+
+Merkle proofs are used to **prove the integrity of opened leaves** in a Merkle tree. A proof ensures that a specific leaf belongs to the committed data by enabling the verifier to reconstruct the **root hash (commitment)**.
+
+A Merkle proof contains:
+
+- **Leaf**: The data being verified.
+- **Index** (leaf_idx): The position of the leaf in the original dataset.
+- **Path**: A sequence of sibling hashes (tree nodes) needed to recompute the path from the leaf to the root.
+
+![Merkle Pruned Phat Diagram](../primitives/merkle_diagrams/diagram1_path.png)
+
+```go
+/// * `leaves` - A slice of leaves (input data).
+/// * `leaf_idx` - Index of the leaf to generate a proof for.
+/// * `pruned_path` - Whether the proof should be pruned.
+/// * `config` - Configuration for the Merkle tree.
+///
+/// # Returns a `MerkleProof` object or eIcicleError
+func GetMerkleTreeProof[T any](
+  mt *MerkleTree,
+  leaves core.HostOrDeviceSlice,
+  leafIndex uint64,
+  prunedPath bool,
+  cfg core.MerkleTreeConfig,
+) (MerkleProof, runtime.EIcicleError)
+```
+
+### Example: Generating a Proof
+
+Generating a proof for leaf idx 5:
+
+```go
+mp, _ := merkletree.GetMerkleTreeProof[byte](
+  &mt,
+  core.HostSliceFromElements(input),
+  5,     /* leafIndex */
+  true, /* prunedPath */
+  core.GetDefaultMerkleTreeConfig(),
+)
+```
+
+:::warning
+The Merkle-path can be serialized to the proof along with the leaf. This is not handled by ICICLE.
+:::
+
+## Verifying Merkle Proofs
+
+```go
+/// * `proof` - The Merkle proof to verify.
+///
+/// # Returns a result indicating whether the proof is valid.
+func (mt *MerkleTree) Verify(mp *MerkleProof) (bool, runtime.EIcicleError)
+```
+
+### Example: Verifying a Proof
+
+```go
+isVerified, err := mt.Verify(&mp)
+assert.True(isVerified)
+```
+
+## Pruned vs. Full Merkle-paths
+
+A **Merkle path** is a collection of **sibling hashes** that allows the verifier to **reconstruct the root hash** from a specific leaf.
+This enables anyone with the **path and root** to verify that the **leaf** belongs to the committed dataset.
+There are two types of paths that can be computed:
+
+- [**Pruned Path:**](#generating-merkle-proofs) Contains only necessary sibling hashes.
+- **Full Path:** Contains all sibling nodes and intermediate hashes.
+
+![Merkle Full Path Diagram](../primitives//merkle_diagrams/diagram1_path_full.png)
+
+To compute a full path, specify `pruned=false`:
+
+```go
+mp, _ := merkletree.GetMerkleTreeProof[byte](
+  &mt,
+  core.HostSliceFromElements(input),
+  5,     /* leafIndex */
+  false, /*non-pruned is a full path --> note the pruned flag here*/
+  core.GetDefaultMerkleTreeConfig(),
+)
+```
+
+## Handling Partial Tree Storage
+
+In cases where the **Merkle tree is large**, only the **top layers** may be stored to conserve memory.
+When opening leaves, the **first layers** (closest to the leaves) are **recomputed dynamically**.
+
+For example to avoid storing first layer we can define a tree as follows:
+
+```go
+mt, err := merkletree.CreateMerkleTree(layerHashers, leafSize, 1 /*min layer to store*/);
+```