Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(evaluate): batched evaluation #346

Open
akotyla opened this issue Feb 11, 2025 · 0 comments
Open

feat(evaluate): batched evaluation #346

akotyla opened this issue Feb 11, 2025 · 0 comments
Assignees
Labels
evals Adding new evaluation pipelines or improving existing ones feature New feature or request

Comments

@akotyla
Copy link
Collaborator

akotyla commented Feb 11, 2025

Feature description

Enhance the _call_pipeline method to support batched evaluation. Instead of processing the whole dataset at once, the method should allow to group dataset items into batches and process them sequentially.

Motivation

APIs like e.g. Azure OpenAI imposes rate limits on the number of requests a user or client can make within a specified period. Currently, _call_pipeline processes the whole dataset at once, which can quickly exceed these limits.

Additional context

No response

@akotyla akotyla added evals Adding new evaluation pipelines or improving existing ones feature New feature or request labels Feb 11, 2025
@akotyla akotyla moved this to Backlog in ragbits Feb 11, 2025
@akotyla akotyla added this to ragbits Feb 11, 2025
@akotyla akotyla self-assigned this Feb 11, 2025
@akotyla akotyla changed the title feat: batched evaluation feat(evaluate): batched evaluation Feb 11, 2025
@akotyla akotyla removed this from ragbits Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
evals Adding new evaluation pipelines or improving existing ones feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant