Sweeper/timeout doesn't work after dynomite connection is re-established

[manjurad]

I have a conductor server along with Dynomite DB that is running a workflow where the client doesn't respond and the timeout kicks in via sweeper functionality the task times out and is re-scheduled.

But if I take the Dynomite server down temporarily and bring it back, then timeout of tasks stop working. When Dynomite server is not reachable I see logs/traceback about error in running sweeper which is as expected, once connection is re-established those errors stops but sweeper doesn't pick up any workflows to sweep.

Based on debug logs and redis commands issued it looks like the unack queue is not picked up after establishing connection. Any ideas/comments on how to fix this?

[kishorebanala]

@manjurad It looks more like a Dyno-queue issue. Please reach out to them directly here: https://github.com/Netflix/dyno-queues

[manjurad]

Thanks I have opened the issue Netflix/dyno-queues#85 here.