Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skip dag instance creation if no new message is available in kafka #700

Closed
kevinwallimann opened this issue Jun 23, 2022 · 0 comments · Fixed by #707
Closed

Skip dag instance creation if no new message is available in kafka #700

kevinwallimann opened this issue Jun 23, 2022 · 0 comments · Fixed by #707
Labels
enhancement New feature or request
Milestone
Sprint 51

Comments

@kevinwallimann
Copy link
Collaborator

kevinwallimann commented Jun 23, 2022 •
edited
Loading

Feature

  • Add logic in Executors (see around lines 100+)
  • For Hyperdrive jobs, determine if there are messages to be consumed, using the service created in Add service to decide if a DagInstance creation can be skipped #690 HyperdriveOffsetComparisonService.isNewJobInstanceRequired
  • Create a HyperdriveExecutor, copy execute method from SparkExecutor, job status must be updated always (cannot be skipped)
  • If no message is available, finish stop with status No Data
  • If message is available, use the SparkExecutor
@kevinwallimann kevinwallimann added the enhancement New feature or request label Jun 23, 2022
@kevinwallimann kevinwallimann changed the title Add HyperdriveNewMessages Sensor - Backend Skip dag instance creation if no new message is available in kafka Jun 23, 2022
@jozefbakus jozefbakus added this to the Sprint 50 milestone Jun 24, 2022
@jozefbakus jozefbakus modified the milestones: Sprint 50, Sprint 51 Jul 20, 2022
@kevinwallimann kevinwallimann mentioned this issue Jul 26, 2022
Merged
This was referenced Aug 3, 2022
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
Sprint 51
Development

Successfully merging a pull request may close this issue.

Feature/700 skip dag instance 2 AbsaOSS/hyperdrive-trigger
2 participants
@jozefbakus @kevinwallimann