Uniqueness in exq jobs

If Exq doesn’t support it out of the box, then I’d look at adding an idempotency key to the job params.

Then create a middleware that will:

1. acquire a distributed lock on the key in the before callback
2. check if the key already exists in a completed_jobs set.
3. Add the key to a completed_jobs set in the after_success callback, using a TTL so the storage doesn’t grow too high.
4. Release the distributed lock in both the after_success and after_fail callbacks.

Redis has support for all the operations required, but there’s probably lots of details to get right, so might be best to try porting an existing solution (from sidekiq?).

Yes Exq doesn’t support out of the box. But I think implementing idempotency in my case is a costly solution. I will simply call some common module’s method for adding jobs to the queue which will make sure that if the same job(with same params) exists in the pending queue/running queue, it will not add to the queue

The code would need to scan the main queue, and the retry queue for each exq worker node.
What if the job is no longer in the queue, but was recently completed?