Skip to content
forked from mgk/rq-retry

Retry support for RQ queuing system.

License

Notifications You must be signed in to change notification settings

EstudioNexos/rq-retry

 
 

Repository files navigation

Build Status Coverage Status Downloads

RQ Retry

RQ Retry is a package that adds retry functionality to the RQ queueing system. It can retry failed jobs immediately or optionally schedule them to retry after a delay using RQ Scheduler.

Installation

pip install rq-retry
pip install rq-scheduler # optional

Usage

Run worker process:

rqworker -w rq_retry.RetryWorker
rqscheduler # optional

The RetryWorker performs work like a regular RQ Worker. In addition it periodically checks for jobs on the failed queue and requeues them to be run again. If a job fails too many times it is moved to the dead letter queue and will not be tried again.

Options

RQ Retry can be configured with environment variables.

Name Default Description
RQ_RETRY_MAINT_INTERVAL 30 How often, at most, to check for failed jobs. The check is not guaranteed to happen every RQ_RETRY_MAINT_INTERVAL seconds. See Maintenance Tasks.
RQ_RETRY_MAX_TRIES 3 Maximum number of time a job may be attempted. This includes the first time the job was run. So a value of 3 will try the job 2 times. A value of 1 or less disables retry. Zero has no special meaning.
RQ_RETRY_DELAYS 5 How long to delay each job retry in seconds. This can be a single float value or a comma separated list of floats. A simple exponential backoff can be achieved by using a value like 3,10,60. This causes the first retry of each job to be delayed 3 seconds, the second retry 10 seconds, and the third and subsequent retries to be delayed 60 seconds. To disable, set to any non numeric value like disabled. When delays are disabled jobs are requeued immediately without using RQ Scheduler.
RQ_RETRY_DEAD_LETTER_QUEUE dead_letter_queue name of dead letter queue

Maintenance Tasks

rqworker is a command line wraper around an RQ Worker. It configures a Worker instance and then calls its work() method which loops indefinitely looking for jobs and running them. Workers also perform maintenance tasks periodically in their work loop. In the case of RetryWorker maintenance tasks include retrying failed jobs.

This means that RetryWorker failed job handling is only performed when the RetryWorker wakes up to handle a "regular" job from a queue. For this reason it is suggested that RetryWorker be used as a drop in replacement for Worker. In an active system this will ensure that failed jobs are retried promptly.

Developing

Pull requests welcome.

Getting started:

make install-dev
make test

Note for conda users

If you're using conda some tox envs may not be able to run tests. To solve this update your PATH so that no python, python3, etc. points at a Python managed by conda. For example on OS X:

export PYTHON_ROOT="/Library/Frameworks/Python.framework/Versions/3.4"
export PATH="$PYTHON_ROOT/bin:$PATH"
make install-dev
make test

License

MIT License

About

Retry support for RQ queuing system.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 95.7%
  • Makefile 4.3%