Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add dpl command for foreachBatch execution #83

Closed
kortemik opened this issue Oct 22, 2024 · 0 comments
Closed

add dpl command for foreachBatch execution #83

kortemik opened this issue Oct 22, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@kortemik
Copy link
Member

Description
add dpl command for enforced batch/driver execution. currently aggregations are processed in paralel mode which does not permit all the possible use cases that spark offers.

| teragrep exec foreachBatch [|transformStatements]

Use case or motivation behind the feature request
there are cases where the aggregation should happen only in sequential mode and forget the previous batch. such are when only batch-window aggregations are required like for example supporting a use case where trigger is configured for 5 minutes and the data between now and the last 5 minutes would be required to do the aggregation.

Related issues

Additional context

@kortemik kortemik added the enhancement New feature or request label Oct 22, 2024
@kortemik kortemik changed the title add dpl command for enforced batch/driver/sequentialMode execution add dpl command for foreachBatch execution Oct 22, 2024
@eemhu eemhu self-assigned this Oct 22, 2024
@q22u q22u closed this as completed Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants