1. Installation

Installation

Aphrodite is primarily a python library, but also contains CUDA C/C++ code that needs to be compiled. We provide pre-compiled binaries in a wheel, so users don't have to deal with the compile times.

Requirements

1. Linux, Windows (WSL)
2. Python 3.8 - 3.11
3. NVIDIA (compute capability > 6.1) and AMD (MI200+, NAVI)

Supported GPUs

Any NVIDIA GPU with a compute capability of 6.1 or higher. Refer to this page for a full list of CUDA GPUs:

https://developer.nvidia.com/cuda-gpus.

Or, you can manually find out your GPU's Compute Capability by opening a Python interpreter and running:

>>> import torch    # if you don't have `torch` installed, run `pip install torch` first
>>> print(torch.cuda.get_device_capability())

This should print something like this: (7, 5), which would indicate a CC of 7.5

If you do not meet the minimum CC, you will not be able to run Aphrodite. At the moment, compute capability of 7.5 or higher is required for AWQ quantization scheme; you can use GPTQ if your GPU does not support it

Pre-compiled binaries via PyPi

For the pip package, all you need is NVIDIA drivers.

$ pip install -U aphrodite-engine --extra-index-url https://downloads.pygmalion.chat/whl

Build from source

Caution

Ninja during build uses 8GB of RAM per thread. If you're running out of memory, export MAX_JOBS=1, or the highest number you can do before you OOM. e.g., if you have 32 threads but 64GB of RAM, limit MAX_JOBS to 8.

The embedded runtime sets up micromamba and downloads CUDA 12.2 along with all the required dependencies.

$ git clone https://github.com/PygmalionAI/aphrodite-engine.git && cd aphrodite-engine
$ ./update-runtime.sh  # this can take a while

You can then use the runtime.sh script to enter the Aphrodite Engine environment. You may also simply prefix all your Aphrodite-related commands with ./runtime.sh to execute them within the created env.

Alternatively, if you already have an environment with CUDA 12, you can simply run the following to build aphrodite-engine:

$ pip install -e .  # this can take a while

Installation on Windows

Credits to @pyroserenus for the initial draft of the guide.

Windows is currently not supported, but you can use WSL2 to install Aphrodite Engine.

1. Open Windows PowerShell as administrator.
2. Type wsl --install and hit enter. This may take a while.
3. It will ask for a username and password. These are arbitrary.
4. Reboot your PC.
5. Run this command:

sudo apt update && sudo apt upgrade -y && sudo apt install python3 python3-pip git wget curl bzip2 tar

The above command will update all installed packages on WSL, then install python.

6. Install Aphrodite Engine. You can now install Aphrodite Engine via pip. If that fails, try building from source.