Persistent Producer Event-buffer?

[mashton]

@hlx

I worked on a simlar system and what we did was:

1. store the data received from external provider(s) into our own database
2. have a producer produce the data from the database as events
3. have a producer_consumer format the data
4. have consumers handle the data

Because we had the data in our own database we could “stop” the server and continue later (deploy, etc)

This is one of the patterns we have in mind, but where we get caught up is the connection between the database and the producer. Can you elaborate on how the producer is notified of new data?

We’ve tried out a PoC using RabbitMQ pub/sub where the flow is:

1. ConsumerSupervisor worker finishes a batch
2. Worker publishes a new “batch event” (based on data from the previous batch) to a message queue
3. Producer, subscribed to that message queue, receives a notification and produces a new event
4. Start over

This sounds nice, but it feels like we’re side-stepping a lot of the benefits of using Elixir for this sort of problem. The bottleneck becomes the messaging queue