Limit number of retries

[tremby]

When a task fails (like an unhandled exception) it is thrown in the failing jobs list but also retried. It doesn't seem there's any limit to the number of retries, and that means a job which is doomed to failure (due to my bad programming) continues to fail all day. I get a message sent to Rollbar each time, and that adds up to a lot of messages and eats my quota.

Is there a way to set a retry limit?

[manuganji]

Is there a work around for this?

[DEKHTIARJonathan]

Is there any update on this ? I have the exact same issue ...
@tremby @manuganji did you find a workaround ?

[tremby]

I honestly don't recall, but I don't think so. Sorry!

[manuganji]

No, I haven't found it. I just fixed that scenario where my task was failing and made sure that I covered all the scenarios that I could predict. An ugly workaround is to wrap your whole task code in a try except block and only throw known exceptions and absorb all others. Of course, this only works if you have limited task types.

[Eagllus]

One way I resolved this issue is with a function called retry
This is designed to work with iLO's that are unreliable with there response/connections.

def retry(func, *args, **kwargs):
    """
    a custom retry for setting iLO information
    """
    count = kwargs.get('count', 0)
    max_retries = kwargs.get('max_retries', 3)
    countdown = kwargs.get('countdown', 30)
    exc = kwargs.get('exc', BaseException)

    if count < max_retries:
        time.sleep(countdown)
        count += 1
        func(*args, count=count, max_retries=max_retries, countdown=countdown)
    else:
        raise exc

I used it like this

def set_host_power(ilo, values, **kwargs):
    try:
        return ilo.call_ilo('set_host_power_saver', values['host_power_saver'])
    except (IloCommunicationError, socket.timeout, socket.error) as exc:
        return retry(set_host_power, ilo, values, exc=exc, **kwargs)

This will make allow retry to retry it an max_retries (default: 3) before raising the exception.
You could instead return a state instead of raising the exception.

[dangerski]

If you are using AWS SQS, my workaround is to make a deadletter queue and then on your task queue enable "use redrive policy" which "Sends messages into a dead letter queue after exceeding the Maximum Receives." Then set the maximum receives to the amount of retries you want to allow.

[pilgrim2go]

@dangerski : I have the same problem but even if I configure AWS Queue to use redrive. It doesn't help.
Diving into code to see why.

[mm-matthias]

The ack_failure/ack_failures options do not work for tasks that fail due to timeout. After the worker is killed, tasks remain in the queue. Once the worker is reincarnated, it fetches the task, runs into timeout and the whole things starts all over again.

[Waszker]

@mm-matthias I have exactly the same problem. Have you found the solution for it? It there a way to specify number of retries before termination?

[mm-matthias]

@Waszker We are using celery/redis instead of django-q for more than a year, so I don't have a solution for this problem

[Waszker]

@Waszker We are using celery/redis instead of django-q for more than a year, so I don't have a solution for this problem

That'a a pity... Thanks for the answer though...

[timomeara]

i have a PR for a retry limit
#466
try it out :)

[abhishek-compro]

This can be closed