This application demonstrates a Multimodal Retrieval-Augmented Generation (RAG) system using the Qwen2-VL model and a custom RAG implementation. It allows users to upload images and ask questions about them, combining visual and textual information to generate responses.
It is deployed here on HuggingFace Spaces https://huggingface.co/spaces/clayton07/qwen2-colpali-ocr
- Python 3.8+
- Pytorch 2.4.1
- Torchvision 0.19.1
- Qwen V1
- Byaldi
- CUDA-compatible GPU (recommended for optimal performance)
-
Clone the repository:
git clone https://github.com/Claytonn7/qwen2-colpali-ocr.git cd multimodal-rag-app -
Create a virtual environment:
python -m venv venv source venv/bin/activate # On Windows, use `venv\Scripts\activate` -
Install the required packages:
pip install -r requirements.txt
-
Ensure you're in the project directory and your virtual environment is activated.
-
Run the Streamlit app:
streamlit run app.py -
Open a web browser and navigate to the URL provided by Streamlit (usually
http://localhost:8501).
- Choose to upload an image or use the example image.
- If uploading, select an image file (PNG, JPG, or JPEG).
- Enter a single keyword in the provided input field.
- Adjust the maximum number of tokens for the response using the slider.
- View the extracted text from the image, with the searched keyword highlighted. Example screenshot here
NB: Check the examples-app directory on this repo for more example screenshots.
The app utilizes the free tier of HuggingFace Spaces, which only has support for CPU, resulting in very slow processing times. For optimal performance, it's recommended to run the app locally on a machine with GPU support.
- This project uses the Qwen2-VL model from Hugging Face.
- The byaldi implementation of the colpali model.
Contributions are welcome! Please feel free to submit a Pull Request.
This project is licensed under the GPL-2.0 License