Restarting rq-scheduler reschedules all periodics #4302
Conversation
| interval = kwargs['interval'] | ||
| if isinstance(interval, timedelta): | ||
| interval = interval.seconds | ||
| interval = int(interval.total_seconds()) |
There was a problem hiding this comment.
What's the diff between total_seconds() and seconds?
There was a problem hiding this comment.
timedelta(days=1).seconds == 0
redash/schedule.py
Outdated
| def job_id(kwargs): | ||
| metadata = kwargs.copy() | ||
| if 'func' in metadata: | ||
| metadata['func'] = metadata['func'].__name__ |
There was a problem hiding this comment.
Can there be a job definition without func? Also, isn't the func name good enough as a job id?
There was a problem hiding this comment.
Not sure what I was thinking with checking for existence in metadata. Anyway, removed in b5237e7.
Hashing all scheduling details (instead of using the job name as an id) allows the scheduler to pick up any changes in schedule and reschedule jobs accordingly. For example - say you want to update the interval or results_ttl for a specific job - if you schedule with a job name, it'll be more difficult to pick up those changes and reschedule the job.
There was a problem hiding this comment.
How about the job id becomes metadata['func'].__name__:hash? This way when poking around and seeing a job id it's easy to tell what job it references?
|
|
||
| manager = AppGroup(help="RQ management commands.") | ||
|
|
||
|
|
||
| @manager.command() | ||
| def scheduler(): | ||
| schedule_periodic_jobs() | ||
| jobs = periodic_job_definitions() | ||
| schedule_periodic_jobs(jobs) |
There was a problem hiding this comment.
Why not call the function that returns job definitions in schedule_periodic_jobs?
There was a problem hiding this comment.
Purely for testability.
| interval = kwargs['interval'] | ||
| if isinstance(interval, timedelta): | ||
| interval = interval.seconds | ||
| interval = int(interval.total_seconds()) |
When
rq-schedulerrestarts, it deletes all existing scheduled jobs and reschedules them. The result of this is that, for example, tasks that should run daily may run more than once a day.Instead of deleting all jobs and rescheduling them, the
schedulercommand should respect previously scheduled jobs and continue their original iterations.