GenStage - How to trigger demand from consumer

Questions / Help
,
1

Im using GenStage for a project. The producer is a Job Queue, and the consumer is a ConsumerSupervisor that runs those jobs. I have placed those stages in my Supervision tree. However, I realised that when the app starts up, there arent any jobs in the queue, so when the consumer demands the jobs, there are 0. Thus, the consumer stops demanding jobs since 0 jobs are given to the consumer. So jobs that get enqueued later on will never get demanded.

I have looked up manual demand as a possible solution, where I would just use a manual subscription and call GenStage.ask every 5 seconds or so from the consumer. However, this feels kind of hacky, and I will have to reimplement the default demanding system which is pretty smart IMO.

I also thought that I could use GenStage.cast to sort of “trigger” the consumer to demand something from the producer, but I realised that it didnt do so.

What is the best way to go about solving this?

Thanks,
Ravern

3 Likes
2

This is a pretty old topic, but in case you’re still interested in this, the docs suggest buffering demand in the producer if the producer cannot satisfy the demand: https://hexdocs.pm/gen_stage/0.14.0/GenStage.html#module-buffering-demand

edit: I totally read it as September 17th, not September 2017

2 Likes
3

:joy:Thanks!

4

This is a common issue with job queues. I have a Sidekiq Enterprise compatible package called Kiq, where I faced the same dilemma. You can see the producer implementation here: https://github.com/sorentwo/kiq/blob/master/lib/kiq/queue/producer.ex

A few things to note:

  • it fetches jobs immediately on init
  • fetching jobs schedules another fetch in the near future
  • it uses buffered demand to keep consumers satisfied

According to the bench script the system is able to handle (enqueue and process) about 25,000 jobs per second with this approach.

5

As an aside, this is the first I’ve written about Kiq. It is under wraps while the pro/enterprise feature set is built out—but it is used in production to process tens of millions of jobs.

Building a job processing library is a great learning experience, but if you have an immediate business need I would recommend one of the existing solutions.