Way to remove lock in application code

[disbelief]

I recently upgraded to 4.0.8 from an older version (2.7.0).

My problem is that I had implemented a dont_perform(jid, interval, *args) worker method, which complements Sidekiq's perform_at and perform_in scheduler methods. Essentially it just removes the a scheduled job from the queue so that it doesn't run when it was originally scheduled.

My code also removed any uniqueness lock that might have been present for that job. However it doesn't work anymore. This bit of code is what I had previously:

module Worker
  module ClassMethods
    def dont_perform(jid, interval, *args)

      # (code to remove scheduled job from sidekiq's queue)

      if self.sidekiq_options_hash['unique']
        unique_job_id = SidekiqUniqueJobs::PayloadHelper.get_payload(self.to_s, self.sidekiq_options_hash['queue'], args)
        Sidekiq.redis do |conn|
          conn.watch(unique_job_id)
          if conn.get(unique_job_id).to_i == 2
            conn.multi do |multi|
              multi.del(unique_job_id)
            end
          else
            conn.unwatch
          end
        end
      end
    end
  end
end

I'd love it if there was a method I could call within SidekiqUniqueJobs to remove the lock so that I don't need to write my own redis commands to do so.

Wondering if anyone can point me in the right direction on how to do this properly?

[mhenrixon]

I will have a look today hopefully have some details for you in an hour or two.

[mhenrixon]

How are you using the the dont_perform method? Could you provide an example?

[disbelief]

Sure, here's a somewhat contrived one:

job_time = Time.now + 1.hour
job_id = ScheduledWorker.perform_at(job_time, some_arg1, some_arg2)

# ... then some time later, before the job has executed:

ScheduledWorker.dont_perform(job_id, job_time, some_arg1, some_arg2)

[disbelief]

Sorry, in the above example you can assume that ScheduledWorker would have a uniqueness setting of :until_executing, :until_executed, or :until_and_while_executing.

[mhenrixon]

I have started something. Version 4.0.9 has the possibility of removing keys by pattern through command line and console. One thing I was considering for that part was to ever only remove keys where the jid doesn't exist in the queue the job belongs to. Would you mind sharing how you delete the job from the scheduled queue? Might help me complete this.

[disbelief]

@mhenrixon sorry for the delay. Here's my full dont_perform code that includes removing the job from the queue:

def dont_perform(jid, interval, *args)
  int = interval.to_f
  ts = (int < 1_000_000_000 ? Time.now.to_f + int : int)
  Sidekiq::Client.rem jid, 'class' => self, 'args' => args, 'at' => ts

  if self.sidekiq_options_hash['unique']
    unique_job_id = SidekiqUniqueJobs::PayloadHelper.get_payload(self.to_s, self.sidekiq_options_hash['queue'], args)
    Sidekiq.redis do |conn|
      conn.watch(unique_job_id)
      if conn.get(unique_job_id).to_i == 2
        conn.multi do |multi|
          multi.del(unique_job_id)
        end
      else
        conn.unwatch
      end
    end
  end
end

I wrote this code over 2 years ago so it's not very fresh in my mind. Let me know if this helps at all.

Perhaps, if you'd just expose a way for me to remove the unique lock for the job, then I could handle the rest of the logic in my own application code? Feels like actually removing the job from the queue is outside the scope of sidekiq-unique-jobs's responsibilities.

[mhenrixon]

There is a way but due to the digest of the unique arguments I suppose it will be easier to handle it from the actual worker instance.

For now it is only possible to list and remove all jobs. There is both a console and a command line version of doing it. Check the docs while I think of how to simplify this a little.

[mhenrixon]

Specifically I think there should be numerous ways of removing/unscheduling/clearing jobs.

1. Removing every key matching some criteria (already done and released)
2. Removing a key based on arguments provided to the job. The trick here is to make the API real easy to use.
3. Removal by JID. This one is tougher since I might have to query sidekiq for the message|item|job.

[mhenrixon]

I'm still toying with this. Think I might have a solution soon, a little trickier than I thought it would be.

[disbelief]

No problem of course! Let me know if there's anything I can do to help.

[Slania]

@mhenrixon it seems like the request is to just drop the uniqueness lock for some set of inputs right? The gem shouldn't actually have to deal with anything to do with the job itself, just with releasing the redis lock.

So the API would look something like:

SidekiqUniqueJobs.unlock!(jid)
SidekiqUniqueJobs.unlock!(worker, args)

What am I missing?

[disbelief]

Had a little time to dig into the sidekiq-unique-jobs source and came up with a way to do this:

module Sidekiq
  module Worker
    module ClassMethods

      def dont_perform(jid, *args)

        # remove the job from sidekiq's scheduler:
        schedule = Sidekiq::ScheduledSet.new
        job = schedule.find {|j| j.klass == self.name && j.jid == jid }
        job.delete if job

        # remove any unique lock for the scheduled job
        if self.sidekiq_options_hash['unique']
          item = { "class" => self.name, "args" => args }.merge(self.sidekiq_options)
          unique_key = SidekiqUniqueJobs::UniqueArgs.digest(item)
          SidekiqUniqueJobs::Unlockable.unlock_by_key(unique_key, jid)
        end
      end

    end
  end
end

This requires that you remember the jid for any scheduled job you would like to be able to cancel. I was already doing this so it wasn't a big deal for me.

I think ideally the method signature would be dont_perform(interval, *args). However I couldn't get any of the other SidekiqUniqueJobs::Unlockable unlocking methods that don't take a jid to work.

[mhenrixon]

Check SidekiqUniqueJobs::Utils and bundle exec jobs console