We’ve recently started needing durable background jobs and we seem to be taking an approach I haven’t really seen discussed. Rather than relying on a job queuing library we use straight OTP and rely on application state to handle durability.
What I mean by “rely on application state to handle durability” is rather than having some sort of queuing system like a “jobs” table or a separate Redis server we just query our data and depending on the state of the data we can determine if we need to start a job.
An example of this is we need to register a webhook with a 3rd party API for our application to work. When we register the webhook we receive a webhook id back from the 3rd party API and store that in our application. Suffice it to say it’s really important that these webhooks get registered. We need to be able to handle failure states (retries) and recover from system restarts (durability).
When we run the process to create an account we start a “worker” GenServer that is responsible for registering the webhook. If all goes well the webhook is registered, we store the id and the GenServer stops. If things don’t go to plan we try again after a period of time using Process.send_after/4.
When the system restarts we have a “booter” GenServer that is responsible for finding all accounts that don’t have a webhook id and starting worker processes to get those accounts taken care of.
Using this methodology we stay completely within the ecosystem of OTP and our application. We don’t have to introduce any new dependencies or worry about our job queue and application state getting out of whack. OTP has all the tools necessary to make scheduling, retries, etc. a breeze.
Curious to hear people’s thoughts on this approach. I’m sure it doesn’t take care of all use cases but it’s worked well for us thus far.
