while_executing has problems at low concurrency
#384
Comments
|
@danpolyuha thank you so much for the detailed report! I will try and create some failing test cases for this scenario. |
|
Using the latest released version I can't replicate the problem @danpolyuha. Running 2 sidekiq worker processes with 25 workers each it seems to produce the expected results. Can you check your gem version, possibly upgrade if not on latest and either close the issue or provide some more details on your setup? This is what my log looks like |
|
@mhenrixon , and check if everything still works fine on your side? If we still can't reproduce it on your side, can you please describe very briefly the logic behind locks in |
|
@danpolyhua if you coul check the configuration I tested with and let me know what is different that would be awesome: |
|
What operating system and redis server version are you on? It could be that your redis server version has trouble with some Lua script feature maybe? |
|
I've tried with a various number of sleeps but they all yield the expected results |
|
@mhenrixon , I see that you're testing everything in Rails. Can you please try to run it in vanilla environment with nothing but Sidekiq and sidekiq-unique-jobs? I have a folder with only two files: require 'sidekiq'
require 'sidekiq-unique-jobs'
class Job
include Sidekiq::Worker
sidekiq_options lock: :while_executing,
lock_timeout: nil,
on_conflict: :reschedule
def perform
puts 'hello'
sleep 1
puts 'bye'
end
end
require_relative 'job'
25.times { Job.perform_async }Then I'm running Sidekiq: Maybe your |
|
It seems that by default sidekiq has concurrency 10 and I have it set to 25 in my Thank you for the investigation. |
mhenrixon
changed the title
while_executing has problems at low concurrency
|
Great news. |
|
Do you have any ideas on how to replicate this in a test? |
|
Not really, unfortunately. |
|
I believe it can be somehow replicated on the level of unit tests close to Lua scripts, but I don't have enough understanding about how your system works to do it. |
|
Fascinating... I actually started writing tests against the Lua scripts yesterday |
|
I'm experiencing the same thing. I believe it's directly tied to the Sidekiq |
|
Which is the opposite of what I would expect, c'est la vie. Anyway, I'm working on fixing this problem once and for all. |
|
Hi @mhenrixon, |
|
I sure am @danpolyuha. I believe it has been fixed already but the fix will be in V7 which is close to ready but not completely done yet. If you are feeling adventurous you could give the master branch a try and see if that fixes your problem. |
|
@danpolyuha there has been numerous releases of |
Describe the bug
When lock type
while_executingis used withlock_timeout: nil, sometimes the system fails to lock the jobs properly and jobs are being executed in parallel.Expected behavior
The jobs never get executed in parallel.
Worker class
Additional context
Hi Mikael,
Thank you very much for your amazing work on this gem.
I believe I might have run into an issue here and would like to verify it with you.
I'm running the given worker using something like this:
Now, since the job must be uniq while executing, "hello" and "bye" must always follow each other. But here is a part of what I get in sidekiq output:
As you can see, jobs are being executed in parallel from time to time.
I'm wondering, is this a bug or have I misconfigured/misunderstood something?
If it is a bug, I believe it must be the scenario when thread 1 checks whether Redis contains the lock and at this moment (before thread 1 writes the digest into Redis) thread 2 starts execution and checks the same.
Please note that everything works properly with
lock_tomeout: 0.The text was updated successfully, but these errors were encountered: