You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After switching to --active-only on jobs that have a large number of dynamic queues, we notice that we start spending a lot of money on GetQueueAttributes calls:
This makes sense, comparing the --active-only API call complexity with the base case, when a is high, then so are the calls:
Context
Calls
Details
qdone worker (while listening, per listen round)
n + (1 per n×w)
w: --wait-time in seconds n: number of queues
qdone worker (while listening with --active-only, per round)
2n + (1 per a×w)
w: --wait-time in seconds a: number of active queues
However the state of the active queues is very cacheable, especially if queues tend to have large backlogs, as ours do.
I propose we add three options:
--cache-url that takes a redis://... cluster url [no default]
--cache-ttl-seconds that takes a number of seconds [default 10]
--cache-prefix that defines a cache key prefix [default qdone]
The presence of the --cache-url option will cause the worker to cache GetQueueAttributes for each queue for the specified ttl. Probably can use mget for this, if we're careful about key slots.
The text was updated successfully, but these errors were encountered:
After switching to
--active-only
on jobs that have a large number of dynamic queues, we notice that we start spending a lot of money on GetQueueAttributes calls:This makes sense, comparing the
--active-only
API call complexity with the base case, when a is high, then so are the calls:qdone worker
(while listening, per listen round)--wait-time
in secondsn: number of queues
qdone worker
(while listening with--active-only
, per round)--wait-time
in secondsa: number of active queues
However the state of the active queues is very cacheable, especially if queues tend to have large backlogs, as ours do.
I propose we add three options:
--cache-url
that takes aredis://...
cluster url [no default]--cache-ttl-seconds
that takes a number of seconds [default 10]--cache-prefix
that defines a cache key prefix [default qdone]The presence of the
--cache-url
option will cause the worker to cacheGetQueueAttributes
for each queue for the specified ttl. Probably can usemget
for this, if we're careful about key slots.The text was updated successfully, but these errors were encountered: