We have events (structs) which we wish to use to update a database table (read model) asynchronously.
The event need to be processed in the order they are published.
Currently we are using Oban, the event is received and a job created on a queue which has a concurrency of 1.
This ensures only one job is picked up at once, thus the jobs are processed in the order they are enqueued.
The one problem we have found is if a job raises and needs to retried then Oban will pull another job off the queue, thus breaking the order. So if the database was to go down or we introduced a bug then we effectively get the events processed in a random order due to the retries.
Is there anything in Oban (including the Pro version) that could help with this?
If not I will explore some other options, to keep Oban I might look at introducing a distributed lock, e.g. using CacheEx, or a persistent queue which runs inside the BEAM, if such a thing exists.
I’d be interested to know if anyone has tackled something similar.
Many thanks!