use lua script to schedule jobs #427
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The old approach had two issues * There was no limit on the zrangebyscore call, this could potentially cause memory issues if there are lot of jobs scheduled within the current time range. * The dequeue and enqueue operation is not atomic. The job could be lost if exq failed to enqueue the job after the zrem command either due to shutdown or redis connection error The new approach uses lua script to run the dequeue and enqueue atomically. A configurable page limit is used to avoid blocking the redis server for long time.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The old approach had two issues
There was no limit on the zrangebyscore call, this could potentially
cause memory issues if there are lot of jobs scheduled within the
current time range.
The dequeue and enqueue operation is not atomic. The job could be
lost if exq failed to enqueue the job after the zrem command either
due to shutdown or redis connection error
The new approach uses lua script to run the dequeue and enqueue
atomically. A configurable page limit is used to avoid blocking the
redis server for long time.
fixes #318 #315