Welcome to the BentoCrewAI project. This project demonstrates how to serve and deploy a CrewAI multi-agent application with the BentoML serving framework.
See here for a full list of BentoML example projects.
This project is a reference implementation designed to be hackable. Download the source code and use it as a playground to build your own agent APIs:
Download source code:
git clone https://github.com/bentoml/BentoCrewAI.git
cd BentoCrewAI/src
Ensure you have Python >=3.10 <=3.13 installed on your system. Install dependencies:
# Create virtual env
pip install virtualenv
python -m venv venv
source ./venv/bin/activate
# Install dependencies
pip install -r requirements.txt --no-deps
Set your OPENAI_API_KEY
environment variable:
export OPENAI_API_KEY='your_openai_key'
./venv/bin/bentoml serve bento_crew_demo.service:CrewAgent
curl -X POST http://localhost:3000/run \
-H 'Content-Type: application/json' \
-d '{"topic": "BentoML"}'
The /run
API endpoint takes the "topic" input from client, and returns the final results.
Use the /stream
endpoint for streaming all intermediate results from the Crew Agent for full context of planning and thinking process:
curl -X POST http://localhost:3000/stream \
-H 'Content-Type: application/json' \
-d '{"topic": "Model Inference"}'
Make sure you have Docker installed and running. Build a docker container image for deployment with BentoML:
bentoml build . --version dev
bentoml containerize crew_agent:dev
Follow CLI output instruction to run the generated container image. E.g.:
docker run --rm \
-e OPENAI_API_KEY=$OPENAI_API_KEY \
-p 3000:3000 \
crew_agent:dev
Follow CrewAI docs on how to customize your Agents and tasks.
- Modify
src/bento_crew_demo/config/agents.yaml
to define your agents - Modify
src/bento_crew_demo/config/tasks.yaml
to define your tasks - Modify
src/bento_crew_demo/crew.py
to add your own logic, tools and specific args - Modify
src/bento_crew_demo/main.py
to add custom inputs for your agents and tasks
We recommend using OpenLLM on BentoCloud for fast and efficient private LLM deployment:
# Install libraries
pip install -U openllm bentoml
openllm repo update
# Login/Signup BentoCloud
bentoml cloud login
# Deploy mistral 7B
openllm deploy mistral:7b-4bit --instance-type gpu.t4.1.8x32
Follow CLI output instructions to view deployment details on BentoCloud UI, and copy your deployed endpoint URL.
💡 For other open-source LLMs, try running
openllm hello
command to explore more.
Next, add the following custom LLM definition to the BentoCrewDemoCrew
class, replace with your deployed API endpoint URL:
from crewai import Agent, Crew, Process, Task, LLM
from crewai.project import CrewBase, agent, crew, task, llm
@CrewBase
class BentoCrewDemoCrew():
...
@llm
def mistral(self) -> LLM:
model_name="TheBloke/Mistral-7B-Instruct-v0.1-AWQ"
return LLM(
# add `openai/` prefix to model so litellm knows this is an openai
# compatible endpoint and route to use OpenAI API Client
model=f"openai/{model_name}",
api_key="na",
base_url="https://<YOUR_DEPLOYED_OPENLLM_ENDPOINT>/v1"
)
And modify the config/agent.yaml
file where you want to use this LLM, e.g.:
researcher:
role: >
{topic} Senior Data Researcher
goal: >
Uncover cutting-edge developments in {topic}
backstory: >
You're a seasoned researcher with a knack for uncovering the latest
developments in {topic}. Known for your ability to find the most relevant
information and present it in a clear and concise manner.
+ llm: mistral
BentoML 1.3.x requires opentelemetry-api==1.20.0 while CrewAI requires opentelemetry-api>=1.27.0; You may ignore the dependency resolver issue and proceed with the 1.27 version that CrewAi requires. BentoML team will update the package to support the newer version of opentelemetry libraries.
# Create virtual env
pip install virtualenv
python -m venv venv
source ./venv/bin/activate
# Install CrewAI after BentoML to override conflict dependency versions
pip install -U bentoml aiofiles
pip install -U crewai "crewai[tools]"
# Export dependencies list
pip freeze > requirements.txt
Join the BentoML developer community on Slack for more support and discussions!