Hi everyone!
I’m writing a fairly simple Elixir application which will do the following:
- Read “jobs” from Redis (a job is just an integer), which become “due” at a certain time
- Add a message to RabbitMQ for each job when it’s due
I’ve got a simple version of this application working using infinite recursion, checking Redis for jobs that are due on each iteration and adding the messages to RabbitMQ as required.
It occurred to me, though, that GenStage is perfect for this kind of application. The producer’s job is to read from Redis to generate events when there is demand, and the consumer’s job is to add the messages to RabbitMQ.
The problem I’m having is that there isn’t always enough supply to satisfy the demand. Each job in Redis becomes due at a certain time, and often there will be no jobs that are due at all. In that case, the producer returns an empty list and the consumers stop asking for more.
Firstly, I’d like to check that GenStage is actually suitable for this kind of workflow. If so, I’m wondering what the best way to implement this would be.
I could have a separate process to poll Redis, and notify the producer when there are jobs available, but it seems like I’d be throwing away the advantages of using GenStage in the first place. Also, the internal buffer could fill up and cause the application to crash if the consumer stopped for a long time (if RabbitMQ was down, for example). In that situation, the preferred behaviour would be to not ask Redis for more jobs until the consumer was ready to consume them.
What would the community recommend for this scenario? 
Thanks!
