Barrier synchronization for oban jobs

Questions / Help
, ,
1

I am using oban to create a group of tasks, all of these tasks are linked to the same parent. After all the tasks are complete, I want to add additional information to that parent that the tasks have completed, so a different status can be rendered to the user.

Additionally to this, there is no guarantee on the order and concurrency of executed tasks, some might fail and require a retry, the server might be restarted when a new deploy is done (hence why oban is used in the first place).

I was thinking on a few approaches:

  1. Exectue a query to check if other jobs are complete at the end of every job, if yes update the status. I do have my fears about this not working correctly in concurrent setting, maybe someone knows more on this topic;
  2. Have a background worker that will periodically check number of jobs completed and update the status, this of course implies some potential delays and queries executed, but for my project it is a viable solution;
  3. Monitor the tasks with a process, re-synchronize the process based on data from the database when the server is restarted.

If you have any better idea it would be great to hear a different perspective.

1 Like
2

Maybe the managers aren’t looking to pay but Oban Pro seems to have what you’re looking for: https://oban.dev/docs/pro/1.2.2/Oban.Pro.Workers.Chunk.html

Though I’m not 100% sure. If you add DB updating logic at the end of each job in the chunk, and have another job sniffing the database and waiting for the data structure to say “all related sub-jobs have completed successfully”, then it will likely achieve what you need.

Though now thinking of it, that could be achieved with Oban’s free version as well.

3

Sadly Oban pro is out of the question, there are no resources for such expenses, moreover it seems an overkill for such a simple feature.

So you are in favor of the solution 2? Makes sense as it would avoid having to deal with any race conditions.