-
Notifications
You must be signed in to change notification settings - Fork 802
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: Update the adaptive batching doc #4571
Conversation
Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>
Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, I would like suggest adding a more complex batching example.
Since BentoML only supports one argument for batchable API, in some cases users want to pass multiple arguments, this can be workarounded with a composite input type and a wrapper service:
from __future__ import annotations
from pathlib import Path
import bentoml
from pydantic import BaseModel
class BatchInput(BaseModel):
image: Path
threshold: float
@bentoml.service
class ImageService:
@bentoml.api(batchable=True)
def predict(self, inputs: list[BatchInput]) -> list[Path]:
# inference logic here using the image and threshold
return [input.image for input in inputs]
@bentoml.service
class MyService:
batch = bentoml.depends(ImageService)
@bentoml.api
def generate(self, image: Path, threshold: float) -> Path:
result = self.batch.predict([BatchInput(image=image, threshold=threshold)])
return result[0]
Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>
@frostming Thanks for the explanation. This use case is very helpful and I added another section to explain it. Please take a look. |
What does this PR address?
Fixes #(issue)
Before submitting:
guide on how to create a pull request.
pre-commit run -a
script has passed (instructions)?those accordingly? Here are documentation guidelines and tips on writting docs.