I’ve worked through a few custom producers for a Broadway setup – one of them paginates data out of a Mongo database. This seems to work pretty well in processing the data when needed. I’ve noticed that the process seems to go to sleep after it finishes its work and there’s nothing left to do – this is affected by the :hibernate_after option. The process is still running, but when new records enter the database (i.e. ones that match the query that the custom-producer is running), they aren’t getting picked up. I’m wondering if I need to “wake up” the producer somehow. Is this a thing?
Thanks in advance for any pointers!
1 Like
You cannot effectively hibernate polling producers because the timeout will wake you up sooner than later. So it should work.
A push producer should receive messages once there is new data, so they should automatically wake up too.
So it all should work. My first check would be to verify if the producer is appropriately storing the demand. Sometimes they are implemented with the assumption that the consumer will automatically ask for more data, but demand is never replayed. If you can’t satisfy it now, you must store to ask it later.
Thank José, much appreciated. I had to take a closer look at the broadway_sqs module. I confess that I don’t follow all of it, but I figured out enough of it. I had to add in something like the schedule_receive_messages/1 that calls Process.send_after/3. That caused my implementation to poll the database periodically so it would pick up new records that matched the query. My module is good enough for internal use, but if I can clean it up and add the proper abstractions, it might be a good candidate for an OffBroadway producer.
