Exq difference between enqueue_at and enqueue_in

Questions / Help
1

I’m curious about the differences between enqueue_in and enqueue_at, the Exq docs aren’t too clear on this.

2

I believe enqueue_at: time means "enqueue this job to run at a specific time time. enqueue_in: interval would mean “enqueue to run interval seconds into the future from now”.

1 Like
3

That makes sense, and in both instances will the job run at the set time/interval or will it await until the worker is free? IE Would it take precedence over other running jobs?

I ask because I have an order processing worker which runs jobs at five minute intervals, however some jobs should only be run once or twice a day (at set times ie. enqueue_at: time) but I noticed that when we got to that time of day the system was getting super bogged down to the point where the workers were just failing completely. I had to actually flush and restart the Redis queue after rolling back to get jobs processing again.

So, I’m wondering if there is a way to say “run this job anytime you can after a specific time”.

4

I don’t use Exq myself, but the general way these queueing systems is that the job becomes available to a worker at the set time, but then it does still require that there is a worker free to fetch it.

That basically is how it works. At the time, the job becomes available, and it’ll be processed when a worker is free. How many jobs are you trying to handle?

2 Likes
5

That’s a difficult question, I also don’t generally use Exq, I just have one project I maintain that uses it. From what I can tell in the config the concurrency is set to 3 and then job queues each have their own concurrent values set (max is 3). So I’m not sure if we’d be handling 3 jobs at once or 11 (sum of all concurrencies per queue). When a lot is going on the queues sometimes get up to 2,000 or 3,000 jobs anything more than that tends to happen only when a worker has stopped.

6

I’m not entirely sure how concurrency works with Exq either. How long does each job take to run? 3k jobs in the queue is quite a lot, what kind of jobs are these?

7

So, I’m wondering if there is a way to say “run this job anytime you can after a specific time”.

It works exactly as you described. Enqueue_in/at puts the job in a ZSET with scheduled time as the score. When the current time is > the scheduled time, exq will push the job to the end of the queue. The exact time of execution depends on how many jobs are pending in the queue.

That’s a difficult question, I also don’t generally use Exq, I just have one project I maintain that uses it. From what I can tell in the config the concurrency is set to 3 and then job queues each have their own concurrent values set (max is 3).

Concurrency is handled per queue, the global config is just a fallback value to be used if you don’t specify per queue concurrency.

So I’m not sure if we’d be handling 3 jobs at once or 11 (sum of all concurrencies per queue)

it would be the later

1 Like