Uniqueness in exq jobs

Questions / Help
, ,
1

I am trying to achieve uniqueness in exq jobs if the params and worker of the job are same(For some of the workers). There is one issue already opened in the library(https://github.com/akira/exq/issues/184). How can I acheive uniqueness in exq jobs for some of the workers

2

If Exq doesn’t support it out of the box, then I’d look at adding an idempotency key to the job params.

Then create a middleware that will:

  1. acquire a distributed lock on the key in the before callback
  2. check if the key already exists in a completed_jobs set.
  3. Add the key to a completed_jobs set in the after_success callback, using a TTL so the storage doesn’t grow too high.
  4. Release the distributed lock in both the after_success and after_fail callbacks.

Redis has support for all the operations required, but there’s probably lots of details to get right, so might be best to try porting an existing solution (from sidekiq?).

3

Yes Exq doesn’t support out of the box. But I think implementing idempotency in my case is a costly solution. I will simply call some common module’s method for adding jobs to the queue which will make sure that if the same job(with same params) exists in the pending queue/running queue, it will not add to the queue

4

The code would need to scan the main queue, and the retry queue for each exq worker node.
What if the job is no longer in the queue, but was recently completed?