Deduplication logic for queues

[fugikzl]

I see from here that job struct has unique identifier and it takes id from push request: [link].

I think it would be very useful if we could have a builtin deduplication mechanism. It would ensure that tasks with the same id are not pushed to the queue if they already exist. This possibly can prevent redundant processing and save resources. Implementation of that feature should not be hard, I guess,

[rustatian]

Hey @fugikzl 👋
Priority queue (in our case) is a binary heap algorithm: link, thus checking for every item to deduplicate would cost us O(n), while just inserting is O(logn). So to check this, instead of having O(logn) insertion time, on every item we'd have O(n) to deduplicate (+ you know, fix-up, fix-down) + O(logn) to insert. Very costly for the performance. ATM I implemented the Delete operation but only for the internal GroupID which is used to delete all jobs with the same GroupID usually when the pipeline was stopped.

[fugikzl]

Thank you for clarification! What do you think about the RPC method to check the task existence by ID. With that, php package can be modified and dedup logic can be handled in producer side. Smth like that:

if(! $queue->exists($id) ){
   $queue->dispatch($task->withId($id));
}

[fugikzl]

I started that discussion, because we met with task duplication problem in in-memory queue in our project. Currently, we are implementing solution with locks, but I think that it would be very good, if rr would have some feature like I proposed.

[rustatian]

Most queues (drivers, like SQS for example) have a special deduplication logic with a deduplicationID or something like this. The idea itself looks pretty good, but I think it should be solved via the driver (like amqp, sqs, etc).

[ejaytkm]

I second :). This feature would be nice to have - as SQS.
The idempotency feature by SQS is very useful. Too bad rr doesn't offer it for now :(

I would write my own codebase, but also want to utilize the golang threads & automated metrics. Have to give up something.