Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add scheduling timeout (non-configurable) #24

Merged
merged 2 commits into from
Dec 27, 2023

Conversation

abelanger5
Copy link
Contributor

@abelanger5 abelanger5 commented Dec 23, 2023

Addresses #10 and #11

Adds a non-configurable scheduling timeout of 30 seconds. This can be made configurable in the step declaration eventually.

A quick note on active versus passive scheduling:

  • Active scheduling runs a timer which schedules time outs precisely using a context deadline or the cron library (this is active because it is "always on"). It is performed by the ticker service. This service currently handles job run timeouts and step run timeouts, but does not handle requeueing or scheduling timeouts.
  • Passive scheduling queries values in the database at some configurable interval, and runs an idempotent query against the database to update the state of a job run or step run if it detects that the job or step runs are past a certain timeout.

The proper way to do this is to schedule all timeouts with the ticker for active listening, and passively query all timeouts as a fallback if a ticker goes down. The passive interval sets the error boundaries on the timing - for example, +5 seconds for requeueing timeouts. This means that if a ticker unexpectedly fails or goes down, the passive scheduler can still cancel jobs and step runs within the error boundary. Smaller error boundaries mean more database load.

A future PR will make all timeouts both passive and active, but at the moment requeueing and scheduling timeouts are passive while job run and step run timeouts are active.

@abelanger5 abelanger5 merged commit 4ca785b into main Dec 27, 2023
3 checks passed
@abelanger5 abelanger5 deleted the belanger/add-scheduling-timeout branch December 27, 2023 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants