From dcaa0b4513f773da995073910f98816622ae104c Mon Sep 17 00:00:00 2001 From: ImmanuelSegol <3ditds@gmail.com> Date: Tue, 19 Dec 2023 08:17:30 -0500 Subject: [PATCH] update readme (#303) * refactor * refactor * refactor: add rust docs * Update README.md Co-authored-by: Jeremy Felder * Update README.md Co-authored-by: Jeremy Felder * refactor * Update icicle/README.md Co-authored-by: Jeremy Felder * Update wrappers/rust/README.md Co-authored-by: Jeremy Felder * Update README.md Co-authored-by: Jeremy Felder * refactor * remove --------- Co-authored-by: Jeremy Felder --- README.md | 171 ++++++---------------------------------- goicicle/README.md | 53 ++++++++----- icicle/README.md | 26 +++--- wrappers/rust/README.md | 37 +++++++++ 4 files changed, 113 insertions(+), 174 deletions(-) create mode 100644 wrappers/rust/README.md diff --git a/README.md b/README.md index 4d95f18b5..45ed8dcd2 100644 --- a/README.md +++ b/README.md @@ -43,151 +43,32 @@ ICICLE is a CUDA implementation of general functions widely used in ZKP. ICICLE ### Prerequisites -- [NVCC] (version 12.0 or newer) -- cmake 3.18 and above -- follow [these instructions](https://github.com/ingonyama-zk/icicle/tree/main/icicle#prerequisites-on-ubuntu) -- Any Nvidia GPU +- [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads) version 12.0 or newer. +- [CMake]((https://cmake.org/files/)), version 3.18 and above. Latest version is recommended. +- [GCC](https://gcc.gnu.org/install/download.html) version 9, latest version is recommended. +- Any Nvidia GPU (which supports CUDA Toolkit version 12.0 or above). -If you don't have access to a Nvidia GPU check out [google-colab](#google-colab). If you require more compute power and are looking to build or do research with ICICLE refer to our [grant program][GRANT_PROGRAM]. +> [!NOTE] +> It is possible to use CUDA 11 for cards which dont support CUDA 12, however we dont offically support this version and in the future there may be issues. +### Accessing Hardware -### Steps +If you don't have access to a Nvidia GPU we have some options for you. -1. Define or select a curve for your application; we've provided a [template][CRV_TEMPLATE] for defining a curve -2. Include the curve in [`curve_config.cuh`][CRV_CONFIG] -3. Now you can build the ICICLE library using nvcc +Checkout [Google Colab](https://colab.google/). Google Colab offers a free [T4 GPU](https://www.nvidia.com/en-us/data-center/tesla-t4/) instance and ICICLE can be used with it, reference this guide for setting up your [Google Colab workplace][GOOGLE-COLAB-ICICLE]. -```sh -mkdir -p build -nvcc -o build/ ./icicle/curves/index.cu -lib -arch=native -``` - -### Testing the CUDA code - -We are using [googletest] library for testing. To build and run [the test suite](./icicle/README.md) for finite field and elliptic curve arithmetic, run from the `icicle` folder: - -For testing, ensure the `BUILD_TESTS` option is enabled in cmake. If not, toggle it on by adding `-DBUILD_TESTS=ON` in the cmake configuration command: - -```sh -cmake -S . -B build -DBUILD_TESTS=ON -``` - -Proceed with the following commands: - -```sh -mkdir -p build -cmake -S . -B build -cmake --build build -cd build && ctest -``` - -NOTE: If you are using cmake versions < 3.24 add `-DCUDA_ARCH=` to the command `cmake -S . -B build` - - -### Rust Bindings +If you require more compute and have an interesting research project, we have [bounty and grant programs][GRANT_PROGRAM]. -For convenience, we also provide rust bindings to the ICICLE library for the following primitives: -- MSM -- NTT - - Forward NTT - - Inverse NTT -- ECNTT - - Forward ECNTT - - Inverse NTT -- Scalar Vector Multiplication -- Point Vector Multiplication +### Build systems -A custom [build script][B_SCRIPT] is used to compile and link the ICICLE library. The environment variable `ARCH_TYPE` is used to determine which GPU type the library should be compiled for and it defaults to `native` when it is not set allowing the compiler to detect the installed GPU type. - -> NOTE: A GPU must be detectable and therefore installed if the `ARCH_TYPE` is not set. - -Once you have your parameters set, run: - -```sh -cargo build --release -``` +ICICLE has three build systems. -You'll find a release ready library at `target/release/libicicle_utils.rlib`. +- [ICICLE core][ICICLE-CORE], C++ and CUDA +- [ICICLE Rust][ICICLE-RUST] bindings +- [ICICLE Golang][ICICLE-GO] bindings -To benchmark and test the functionality available in RUST, run: - -``` -cargo bench -cargo test -- --test-threads=1 -``` - -The flag `--test-threads=1` is needed because currently some tests might interfere with one another inside the GPU. - -### Example Usage - -An example of using the Rust bindings library can be found in our [fast-danksharding implementation][FDI] - -### Supporting Additional Curves - -Supporting additional curves can be done as follows: - -Create a JSON file with the curve parameters. The curve is defined by the following parameters: -- ``curve_name`` - e.g. ``bls12_381``. -- ``modulus_p`` - scalar field modulus (in decimal). -- ``bit_count_p`` - number of bits needed to represent `` modulus_p`` . -- ``limb_p`` - number of (32-bit) limbs needed to represent `` modulus_p`` (rounded up). -- ``ntt_size`` - log of the maximal size subgroup of the scalar field. -- ``modulus_q`` - base field modulus (in decimal). -- ``bit_count_q`` - number of bits needed to represent `` modulus_q`` . -- ``limb_q`` - number of (32-bit) limbs needed to represent `` modulus_q`` (rounded up). -- ``weierstrass_b`` - `b` of the curve in Weierstrauss form. -- ``weierstrass_b_g2_re`` - real part of the `b` value in of the g2 curve in Weierstrass form. -- ``weierstrass_b_g2_im`` - imaginary part of the `b` value in of the g2 curve in Weierstrass form. -- ``gen_x`` - `x` coordinate of a generator element for the curve. -- ``gen_y`` - `y` coordinate of a generator element for the curve. -- ``gen_x_re`` - real part of the `x` coordinate of generator element for the g2 curve. -- ``gen_x_im`` - imaginary part of the `x` coordinate of generator element for the g2 curve. -- ``gen_y_re`` - real part of the `y` coordinate of generator element for the g2 curve. -- ``gen_y_im`` - imaginary part of the `y` coordinate of generator element for the g2 curve. -- ``nonresidue`` - nonresidue, or `i^2`, or `u^2` - square of the element that generates quadratic extension field of the base field. - -Here's an example for BLS12-381. -``` -{ - "curve_name" : "bls12_381", - "modulus_p" : 52435875175126190479447740508185965837690552500527637822603658699938581184513, - "bit_count_p" : 255, - "limb_p" : 8, - "ntt_size" : 32, - "modulus_q" : 4002409555221667393417789825735904156556882819939007885332058136124031650490837864442687629129015664037894272559787, - "bit_count_q" : 381, - "limb_q" : 12, - "weierstrass_b" : 4, - "weierstrass_b_g2_re" : 4, - "weierstrass_b_g2_im" : 4, - "gen_x" : 3685416753713387016781088315183077757961620795782546409894578378688607592378376318836054947676345821548104185464507, - "gen_y" : 1339506544944476473020471379941921221584933875938349620426543736416511423956333506472724655353366534992391756441569, - "gen_x_re" : 352701069587466618187139116011060144890029952792775240219908644239793785735715026873347600343865175952761926303160, - "gen_x_im" : 3059144344244213709971259814753781636986470325476647558659373206291635324768958432433509563104347017837885763365758, - "gen_y_re" : 1985150602287291935568054521177171638300868978215655730859378665066344726373823718423869104263333984641494340347905, - "gen_y_im" : 927553665492332455747201965776037880757740193453592970025027978793976877002675564980949289727957565575433344219582, - "nonresidue" : -1 -} -``` - -Save the parameters JSON file under the ``curve_parameters`` directory. - -Then run the Python script ``new_curve_script.py `` from the root folder: - -``` -python3 ./curve_parameters/new_curve_script.py ./curve_parameters/bls12_381.json -``` - -The script does the following: -- Creates a folder in ``icicle/curves`` with the curve name, which contains all of the files needed for the supported operations in cuda. -- Adds the curve's exported operations to ``icicle/curves/index.cu``. -- Creates a file with the curve name in ``src/curves`` with the relevant objects for the curve. -- Creates a test file with the curve name in ``src``. - -Also files from ``./icicle/curves//supported_operations.cu`` should be added individually to ``add_library`` section of [``./icicle/CMakeLists.txt``][CMAKELISTS] - -Testing the new curve could be done by running the tests in ``tests_curve_name`` (e.g. ``tests_bls12_381``). +ICICLE core always needs to be built as part of the other build systems as it contains the core ICICLE primitives implemented in CUDA. Reference these guides for the different build systems, [ICICLE core guide][ICICLE-CORE-README], [ICICLE Rust guide][ICICLE-RUST-README] and [ICICLE Golang guide][ICICLE-GO-README]. ## Docker @@ -198,14 +79,6 @@ docker build -t . docker run --gpus all -it /bin/bash ``` -## Google Colab - -[Colab](https://colab.google/) is a hosted Jupyter Notebook service that requires no setup to use and provides free access to computing resources including GPUS! - -You can easily run ICICLE in Google Colab on a free GPU instance, this is a great option for those who want to get started with ICICLE instantly without any local setup or GPU. - -Follow this [guide][GOOGLE_COLAB_ICICLE] for more details. - ## Contributions Join our [Discord Server][DISCORD] and find us on the icicle channel. We will be happy to work together to support your use case and talk features, bugs and design. @@ -252,14 +125,20 @@ See [LICENSE-MIT][LMIT] for details. [CRV_TEMPLATE]: ./icicle/curves/curve_template/ [CRV_CONFIG]: ./icicle/curves/index.cu [B_SCRIPT]: ./build.rs -[FDI]: https://github.com/ingonyama-zk/fast-danksharding [LMIT]: ./LICENSE [DISCORD]: https://discord.gg/Y4SkbDf2Ff [googletest]: https://github.com/google/googletest/ [HOOKS_DOCS]: https://git-scm.com/docs/githooks [HOOKS_PATH]: ./scripts/hooks/ [CMAKELISTS]: https://github.com/ingonyama-zk/icicle/blob/f0e6b465611227b858ec4590f4de5432e892748d/icicle/CMakeLists.txt#L28 -[GOOGLE_COLAB_ICICLE]: https://github.com/gkigiermo/rust-cuda-colab -[GRANT_PROGRAM]: https://docs.google.com/forms/d/e/1FAIpQLSc967TnNwxZZ4akejcSi4KOUmGrEc68ZZV-FHLfo8KnP1wbpg/viewform +[GOOGLE-COLAB-ICICLE]: https://github.com/gkigiermo/rust-cuda-colab +[GRANT_PROGRAM]: https://medium.com/@ingonyama/icicle-for-researchers-grants-challenges-9be1f040998e +[ICICLE-CORE]: ./icicle/ +[ICICLE-RUST]: ./wrappers/rust/ +[ICICLE-GO]: ./goicicle/ +[ICICLE-CORE-README]: ./icicle/README.md +[ICICLE-RUST-README]: ./wrappers/rust/README.md +[ICICLE-GO-README]: ./goicicle/README.md + diff --git a/goicicle/README.md b/goicicle/README.md index 623b3953a..a3e5e007d 100644 --- a/goicicle/README.md +++ b/goicicle/README.md @@ -1,27 +1,46 @@ -# ICICLE CUDA to Golang Binding Guide +# Golang Bindings -This guide provides instructions on how to compile CUDA code using the provided Makefile, and then how to use the resulting shared libraries to bind Golang to ICICLE's CUDA code. +To build the shared library: -## Prerequisites +To build shared libraries for all supported curves. -To compile the CUDA files, you will need: +``` +make all +``` + +If you wish to build for a specific curve, for example bn254. + +``` +make libbn254.so +``` + +The current supported options are `libbn254.so`, `libbls12_381.so`, `libbls12_377.so` and `libbw6_671.so`. The resulting `.so` files are the compiled shared libraries for each curve. + +Finally to allow your system to find the shared libraries + +``` +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH/ +``` -- CUDA toolkit installed. The Makefile assumes CUDA is installed in `/usr/local/cuda`. If CUDA is installed in a different location, please adjust the `CUDA_ROOT_DIR` variable accordingly. -- A compatible GPU and corresponding driver installed on your machine. +## Running golang tests -## Structure of the Makefile +To run the tests for curve bn254. -The Makefile is designed to compile CUDA files for four curves: BN254, BLS12_381, BLS12_377 and BW6_671. The source files are located in the `icicle/curves/` directory. +``` +go test ./goicicle/curves/bn254 -count=1 +``` + +## Cleaning up -## Compiling CUDA Code +If you want to remove the compiled files -1. Navigate to the directory containing the Makefile in your terminal. -2. To compile all curve libraries, use the `make all` command. This will create four shared libraries: `libbn254.so`, `libbls12_381.so`, `libbls12_377.so` and `libbw6_671.so`. -3. If you want to compile a specific curve, you can do so by specifying the target. For example, to compile only the BN254 curve, use `make libbn254.so`. Replace `libbn254.so` with `libbls12_381.so`, `libbls12_377.so` or `libbw6_671.so` to compile those curves instead. +``` +make clean +``` -The resulting `.so` files are the compiled shared libraries for each curve. +This will remove all shared libraries generated from the `make` file. -## Golang Binding +# How do Golang bindings work? The shared libraries produced from the CUDA code compilation are used to bind Golang to ICICLE's CUDA code. @@ -44,11 +63,7 @@ func main() { Replace `/path/to/shared/libs` with the actual path where the shared libraries are located on your system. -## Cleaning up - -If you want to remove the compiled files, you can use the `make clean` command. This will remove the `libbn254.so`, `libbls12_381.so`, `libbls12_377.so` and `libbw6_671.so` files. - -## Common issues +# Common issues ### Cannot find shared library diff --git a/icicle/README.md b/icicle/README.md index 1a0e3a4a5..290a6ca71 100644 --- a/icicle/README.md +++ b/icicle/README.md @@ -3,17 +3,14 @@ ## Running tests ```sh -mkdir -p build; cmake -S . -B build; cmake --build build; cd build && ctest; cd .. +mkdir -p build; +cmake -DBUILD_TESTS=ON -DCURVE= -S . -B build; +cmake --build build; +cd build && ctest; +cd .. ``` -## Prerequisites on Ubuntu - -Before proceeding, make sure the following software installed: - -1. CMake at least version 3.18, which can be downloaded from [cmake.org](https://cmake.org/files/) - It is recommended to have the latest version installed. -2. [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu) version 12.0 or newer. -3. GCC - version 9 or newer recommended. +The command above will build ICICLE Core and run the ctest. ## Troubleshooting @@ -81,3 +78,14 @@ Otherwise create a symlink. For example, if the CUDA toolkit is installed with a Alternatively, you can replace the old CUDA root with a symlink to the new CUDA installation with the following command: `ln -sf /usr/local/cuda-12.1/ /usr/lib/nvidia-cuda-toolkit/` + +### 8 - Error while loading shared libraries + +`cmake: error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory` + +Make sure `libssl` is installed. + +``` +sudo apt-get update +sudo apt-get install libssl1.0.0 libssl-dev +``` diff --git a/wrappers/rust/README.md b/wrappers/rust/README.md new file mode 100644 index 000000000..6e9e224c0 --- /dev/null +++ b/wrappers/rust/README.md @@ -0,0 +1,37 @@ +### Rust Bindings + +`icicle-core` defines all interfaces, macros and common methods. + +`icicle-cuda-runtime` defines `DeviceContext` which can be used to manage a specific GPU as well as wrapping common CUDA methods. + +`icicle-curves` implements all interfaces and macros from `icicle-core` for each curve. For example `icicle-bn254` implements curve `bn254`. Each curve has its own [build script](./icicle-curves/icicle-bn254/build.rs) which will build the CUDA libraries for that curve as part of the rust-toolchain build. + +## Building a curve and running tests + +Enter a curve implementation. + +``` +cd icicle-curves/icicle-bn254 +``` + +To build + +```sh +cargo build --release +``` + +The build may take a while because we are also building the CUDA libraries for the selected curve. + +To run benchmarks + +``` +cargo bench +``` + +To run test + +```sh +cargo test -- --test-threads=1 +``` + +The flag `--test-threads=1` is needed because currently some tests might interfere with one another inside the GPU.