I am trying to achieve uniqueness in exq jobs if the params and worker of the job are same(For some of the workers). There is one issue already opened in the library(https://github.com/akira/exq/issues/184). How can I acheive uniqueness in exq jobs for some of the workers
If Exq doesn’t support it out of the box, then I’d look at adding an idempotency key to the job params.
Then create a middleware that will:
- acquire a distributed lock on the key in the
before
callback - check if the key already exists in a
completed_jobs
set. - Add the key to a
completed_jobs
set in theafter_success
callback, using a TTL so the storage doesn’t grow too high. - Release the distributed lock in both the
after_success
andafter_fail
callbacks.
Redis has support for all the operations required, but there’s probably lots of details to get right, so might be best to try porting an existing solution (from sidekiq?).
Yes Exq doesn’t support out of the box. But I think implementing idempotency in my case is a costly solution. I will simply call some common module’s method for adding jobs to the queue which will make sure that if the same job(with same params) exists in the pending queue/running queue, it will not add to the queue
The code would need to scan the main queue, and the retry queue for each exq worker node.
What if the job is no longer in the queue, but was recently completed?