Skip to content

Serving LangGraph Agent as REST API with BentoML, optionally with self-host open-source LLMs

Notifications You must be signed in to change notification settings

bentoml/BentoLangGraph

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 
 
 

Repository files navigation

LangGraph Serving with BentoML

This repo demonstrates how to serve LangGraph agent application with BentoML.

See here for a full list of BentoML example projects.

Overview

  • AI Agent Serving: Serving LangGraph Agent as REST API for easy integration
  • Flexible Invocation: Supports both synchronous and asynchronous (queue-based) interactions.
  • Deployment Options: Run locally or deploy to BentoCloud for scalability.
  • LLM Deployment: Use external LLM APIs or deploy open-source LLM together with the Agent API service

This project serves as a reference implementation designed to be hackable, providing a foundation for building and customizing your own AI agent solutions.

Getting Started

Download source code:

git clone https://github.com/bentoml/BentoLangGraph.git
cd BentoLangGraph/

Follow the step-by-step guide for serving & deploying LangGraph agents with BentoML:

Troubleshoot

When running the example code which uses DuckDuckGo search tool, you may run into the following rate limit error:

RatelimitException('https://duckduckgo.com/ 202 Ratelimit')

You may use a different tool from LangChain's pre-built tools list here or create a custom tool.

For example, you can use exa to replace DuckDuckGo for search:

- from langchain_community.tools import DuckDuckGoSearchRun
- tools = [search]
+ from exa_search import retrieve_web_content
+ tools = [retrieve_web_content]

Interested in using LangGraph with other open-source LLMs? Checkout BentoVLLM for more sample code.

Community

Join the BentoML developer community on Slack for more support and discussions!

About

Serving LangGraph Agent as REST API with BentoML, optionally with self-host open-source LLMs

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published