Note: This project is constantly under development. If there are any issues, feel free to submit an issue or pull request and I can try to help!
Otherwise, feel free to fork/clone/copy the repo and make your own changes! I hope with the newly added docker support it will be easier to set this up, and figure out ways to modify from there. I promise a video introduction is coming soon.
One other thing to note is that due to llama cpp no longer actively supporting vision models, the camera functionality has been temporarily removed.
Pi-Card is an AI powered assistant running entirely on a Raspberry Pi. It is capable of doing what a standard LLM (like ChatGPT) can do in a conversational setting. In addition, if there is a camera equipped, you can also ask Pi-card to take a photo, describe what it sees, and then ask questions about that image.
Raspberry Pi - Camera Audio Recognition Device.
Please submit an issue or pull request if you can think of a better way to force this acronym.
Pi-Card runs on your Raspberry Pi.
With a wake word. Once the main.py, the system will listen for your wake word. Once your wake word has been said, you are officially in a conversation. Within this conversation you do not need to constantly repeat the wake word. The system will continue to listen for your commands until you say something like "stop", "exit", or "goodbye". More information on this / customization can be found in the config.py file.
With a button. If you can get your hands on a breadboard, some wires, and a button, using a button to to handle the conversation is a much smoother (in my opinion) way to interact. This is done by pressing the button, and then speaking your command. The button is a simple GPIO button, and can be set up by following the instructions in the main_button.py file.
The chatbot has a configurable memory of the conversation, meaning if you want the assistant to repeat something it said, or elaborate on a previous topic, you can do so. For quicker responses, you can set the memory to a smaller number in the config.py file.
The system is designed to be a fun project that can be a somewhat helpful AI assistant. Since everything is done locally, it will not be as capable, or as fast, as cloud based systems. However, in the past year significant strides have already been made in small LLM models, and it's likely that this will only continue to improve, meaning so too will this project!
The main reason for this is that I wanted to create a voice assistant that is completely offline and how efficient it can be on the relatively inexpensive hardware of a Raspberry Pi. I figure the hardest part of this project was to make it run fast, so if I can get it working like this, something similar could be done on increasingly more powerful hardware while only being faster.
After downloading the repository, installing the requirements, and following the other setup instructions, you can run the main program by running the following command:
python main.pyor
python main_button.pyOnce the program is running, you can start a conversation with the assistant by saying the wake word. The default wake words are "raspberry", "barry", "razbear" (aka things that the transcription might have accidentally picked up), but you can change this in the config.py file to anything you want.If the button version is in place, you can press the button to start a conversation, or interrupt the assistant at any time. Please note that these options, like what the wake word is, what the GPIO button is, are all things that I set up for my own use. Feel free to change them!
To run the project in a docker container, you can use the following command:
sudo docker-compose build
sudo docker-compose upThis is a recent addition, so may not work perfectly. It also only works for the wake-word version, not sure how to get GPIO access passed to the container.
To keep this system as fast and lean as possible, we use cpp implementations where possible. Some examples are whisper.cpp for audio transcription and llama.cpp for the vision capabilities.
Please clone these repositories wherever you like, and add its path to the config.py file.
Once cloned, please follow the setup instructions to get the models running. Some pointers are given below:
To make pi-card a bit more like a real assistant, there are a couple tools it has access to. These are done through tool-bert, a fine-tuned version of BERT deciding when to access external info. More info on how to make a version of this can be found here
The model is easy to install, but to enable tool access, take a look at .env.example file for context on what keys and secrets are necessary.
For whisper.cpp, you will need to follow the quick-start guide in the README.
If you can hook a camera up to your Raspberry Pi, you can enable the vision model. If you have it, you can ask pi-card to snap a photo, and describe what it sees.
This is done by setting the vision_model to vlm in the config.py file. At the same time, you'll also need to download the appropriate models for this, which is Qwen2-VL-2B-Instruct.
Since this model has a dynamic input image token size, by making the snapped photo smaller, we can speed up the inference time. There's a lot of potential for this and using VLMs on the Raspberry Pi. For info on how to download llama.cpp, the quants, and execute the code, please see here.
The hardware setup is quite simple. You will need a Raspberry Pi 5 Model B, a USB microphone, and a speaker.
The USB microphone and speaker can be plugged into the Raspberry Pi's USB ports. The camera can be connected to the camera port on the Raspberry Pi.
I used the following hardware for my setup:
Please note Pi 5's have a new camera port, hence the new camera connector. At the same time, while this project is focused on making this work on a Raspberry Pi 5, it should work on other devices as well.
The camera connector is optional, but if you want to use the camera functionality, you will need to purchase one.
For setting up the GPIO button, I found the first couple minutes of this tutorial great.
Feel free to use your own, this is what worked for me!
Coming soon, but I plan to add notes here on things currently implemented, and what can be done in the future. Some quick notes on it are below. I've started to make a notion board to keep track of things, but it's not complete yet. Check it out here.
- Basic conversation capabilities
- Camera capabilities
- Benchmark response times
- Test overclocking
- Figure out how to speed up whisper times
- Add ability to interrupt assistant, and ask new question
- Use a custom tuned model for the assistant
- Improved tutorials & videos
- Improve external service function model (tool-bert)
- Test when connected to a portable power source
- Create optional model generation using entropix
- Dockerize repo for testing on more devices
- Test in other languages
- Add more external services
