Throttling of queues

[mabeebam]

Hi,
First of all: Thanks for exq, it is great!

Is throttling of queues (e.g. only x job executions per minute/hour per queue) a feature that will be supported by exq in the future? How would one model throttling in exq? Middleware before_work, after_processed_work, after_failed_work?

Any ideas or opinions welcome! Thanks in advance!

Cheers!

[sineed]

@mabeebam You are right, it can be implemented as a new middleware. I think that the middleware should be similar to sidekiq-throttler gem implementation.

[akira]

@sineed @mabeebam 👍 on using middleware, that should be doable in a similar way to the sidekiq-throttler implementation as sineed mentioned.

[mabeebam]

@sineed @akira Thanks for pointing out sidekiq-throttler. I had a look at it and the following issue
gevans/sidekiq-throttler#8 (comment)
pointed out some concerns regarding its design pattern (As I understood it: re-enqueue after some delay as new job), when you have a lot of jobs in a queue that are re-enqueued over and over again.

I also had look at a gem
https://github.com/enova/sidekiq-rate-limiter
that avoids to re-enque a new job but lpushes the existing job to Redis when throttling params are exceeded during fetching. Still you are pushing and pushing to Redis over and over again, though.

The gem
https://github.com/brainopia/sidekiq-limit_fetch
has functionality to pause a queue, which seems a more adequate approach if one has lots of jobs in a throttling queue. No re-enqueue and no lpushes to Redis.

@sineed @akira How would you realize throttling of queues (with 100.000+ jobs) via unsubscribing and subscribing them (scheduled after some time) when certain throttling params are exceeded? Still as middleware? Any other ideas?

Thanks!
Cheers

[akira]

If we wanted to avoid the fetching and re-queuing we can potentially implement some "server-side" middleware that would limit the available queues. Right now it limits it based on free workers, but it could also apply other rules so that those queues never even get popped if they are limited further by the other middleware. Spot where we are limiting it currently: https://github.com/akira/exq/blob/master/lib/exq/manager/server.ex#L246

[wli0503]

I got an implementation more like the sidekiq-throttler. Already pushed to my forked but haven't submit a PR request yet since there are still some polishing and testing need to be done. If you are interested, you can go to https://github.com/wli0503/exq to check it out. @akira If you got some time, can you take a look at the implementation and tell me how you think about it? Thanks a lot for you time and I am looking forward to hear from you!

[akira]

@wli0503 I think for utilities like this I would prefer it to be an external package - and you can include it in your project. Does that work for you?

You can do a separate pull request for the Exq changes in the files:

lib/exq/redis/connection.ex
lib/exq/middleware/pipeline.ex
lib/exq/middleware/job.ex

as those look ok to be in Exq.

[wli0503]

@akira Thank you so much for taking time reviewing my commit! Yes, I can definitely do that. Once I'm ready I will submit a separate PR for modifications into Exq and keep the throttler as an external package like we discussed. Really appreciated!

[StanBright]

Hey @wli0503, I'm wondering, did you have time to polish your implementation?

[wli0503]

@StanBright Not really. After I did the first implementation I realize that It might not suit my needs as intended, so I have to leave it for now.

[mark-d-holmberg]

How would I trigger this middleware from a job?

[ananthakumaran]

Closing this, as we have added the ability to do this on master branch. Examples can be found at https://github.com/ananthakumaran/exq_limit