I’m trying to figure out what would be the best solution for a following task.
There is an Event with SubEvents, information about which is carried by RabbitMQ.
System logic must respect following conditions: Event can go through different states Start -> Active - Finish and can receive multiple updates while in Active state.
An Event is considered successfully processed only if and when all SubEvents are successfully processed.
Now, I wonder if GenStage/Broadway will allow to do something like that:
Consume Event from RabbitMQ
Fetch SubEvents from 3rd party service (preferably in parallel )
Process them independently (network operations will be involved)
3.1. Retry SubEvent processing stage if something went wrong, for example a network error occured.
Acknowledge Event as processed only and when all SubEvents were successfully processed
Prevent race conditions, for example when one of the SubEvents from Create message is still being processed, but there an Update message is already here.
Have 2 queues. One for Events and One for Sub-events.
When you consume from the Events queue, you create an Event record in the database (assuming you use SQL, but you can use any database) storing the Sub-events, their status and other relevant data. You need to store it in the database to keep track of its status and which sub-events have already been processed. After storing this data, publish one message for each sub-event to the sub-events queue.
When you consume from the sub-events queue, you pull the data from the 3rd-party service and process them accordingly. You can have your retry logic here.
Once a sub-event is successfully processed, you update its record in the database and its status to ‘done’
5 .For an Event to be ‘done’, all its sub-events must be ‘done’.
Does that approach make sense?
Regarding your questions:
Consume Event from RabbitMQ
Yes, both can do that. Broadway was built specifically for events.
Fetch SubEvents from 3rd party service (preferably in parallel )
Yes, you can do this in parallel. Will this be via HTTP? Will you be fetching sub-events from multiple endpoints?
Process them independently (network operations will be involved)
3.1. Retry SubEvent processing stage if something went wrong, for example a network error occurred.
I think this may be logic you’d have to write yourself. You can take advantage of re-queueing in rabbitmq.
Acknowledge Event as processed only and when all SubEvents were successfully processed
Looks like you will need to keep track of an Event’s sub-events and its status. I would store this data in a database as I suggested above.
Prevent race conditions, for example when one of the SubEvents from Create message is still being processed, but there an Update message is already here.
This is logic you will have to handle. You need to make sure that the order of events and properly handle this case.
I think a lot (if not all) of what @gjaldon suggested is still valid, because you could keep some minimal track of state in TermStorage. You wouldn’t need an additional substantial DB. Additionally, depending on the shape of things, using TermStorage might prevent race conditions depending on how you cache the pending insert or update (i.e. you’ll be able to know whether you need to requeue the update or even override the insert with new data or something).
This very much depends on the scope and shape of what you’re doing.
)
