Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Update the adaptive batching doc #4571

Merged
merged 3 commits into from
Mar 13, 2024

Conversation

Sherlock113
Copy link
Contributor

@Sherlock113 Sherlock113 commented Mar 12, 2024

What does this PR address?

Fixes #(issue)

Before submitting:

Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>
@Sherlock113 Sherlock113 requested a review from a team as a code owner March 12, 2024 09:01
@Sherlock113 Sherlock113 requested review from frostming and removed request for a team March 12, 2024 09:01
@Sherlock113 Sherlock113 self-assigned this Mar 12, 2024
@Sherlock113 Sherlock113 added the documentation Documentation, tutorials, and example projects label Mar 12, 2024
Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>
Copy link
Contributor

@frostming frostming left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I would like suggest adding a more complex batching example.

Since BentoML only supports one argument for batchable API, in some cases users want to pass multiple arguments, this can be workarounded with a composite input type and a wrapper service:

from __future__ import annotations

from pathlib import Path

import bentoml
from pydantic import BaseModel


class BatchInput(BaseModel):
    image: Path
    threshold: float


@bentoml.service
class ImageService:
    @bentoml.api(batchable=True)
    def predict(self, inputs: list[BatchInput]) -> list[Path]:
        # inference logic here using the image and threshold
        return [input.image for input in inputs]


@bentoml.service
class MyService:
    batch = bentoml.depends(ImageService)

    @bentoml.api
    def generate(self, image: Path, threshold: float) -> Path:
        result = self.batch.predict([BatchInput(image=image, threshold=threshold)])
        return result[0]

Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>
@Sherlock113
Copy link
Contributor Author

@frostming Thanks for the explanation. This use case is very helpful and I added another section to explain it. Please take a look.

@Sherlock113 Sherlock113 requested a review from frostming March 13, 2024 01:57
@Sherlock113 Sherlock113 merged commit d27415a into bentoml:main Mar 13, 2024
4 checks passed
@Sherlock113 Sherlock113 deleted the adaptivebatchingupdate branch March 13, 2024 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Documentation, tutorials, and example projects
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants