Skip to content

Commit

Permalink
Docs for lower smaller models to mps/coreml/qnn (#3146)
Browse files Browse the repository at this point in the history
Summary:
Pull Request resolved: #3146

ghstack-source-id: 223235858

Reviewed By: mcr229, kirklandsign

Differential Revision: D56340028

fbshipit-source-id: ef06142546ac54105ae87007cd82369917a22b3e
  • Loading branch information
cccclai authored and facebook-github-bot committed Apr 19, 2024
1 parent 0800594 commit d47f9fe
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions examples/models/llama2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,16 @@ Please refer to [this tutorial](https://pytorch.org/executorch/main/llm/llama-de
### Android
Please refer to [this tutorial](https://pytorch.org/executorch/main/llm/llama-demo-android.html) to for full instructions on building the Android LLAMA Demo App.

## Optional: Smaller models delegated to other backends
Currently we supported lowering the stories model to other backends, including, CoreML, MPS and QNN. Please refer to the instruction
for each backend ([CoreML](https://pytorch.org/executorch/main/build-run-coreml.html), [MPS](https://pytorch.org/executorch/main/build-run-mps.html), [QNN](https://pytorch.org/executorch/main/build-run-qualcomm.html)) before trying to lower them. After the backend library is installed, the script to export a lowered model is

- Lower to CoreML: `python -m examples.models.llama2.export_llama -kv --coreml -c stories110M.pt -p params.json`
- MPS: `python -m examples.models.llama2.export_llama -kv --mps -c stories110M.pt -p params.json`
- QNN: `python -m examples.models.llama2.export_llama -kv --qnn -c stories110M.pt -p params.json`

The iOS LLAMA app supports the CoreML and MPS model and the Android LLAMA app supports the QNN model. On Android, it also allow to cross compiler the llama runner binary, push to the device and run.

# What is coming next?
## Quantization
- Enabling FP16 model to leverage smaller groupsize for 4-bit quantization.
Expand Down

0 comments on commit d47f9fe

Please sign in to comment.