lcabrini

lcabrini

Long-running queued background processes

A few years ago a built a system in Go that I’m trying to rebuild as a part of learning more about Ash and Phoenix. This involved users being able to generate various reports relating to water bodies in West Africa. We got the data by talking to a remote web service. This was usually very slow and it could take quite a while to get a repsonse back. In addition, we were only supposed to allow a limited number of report requests to run at a time (a value that they admins could change). Once we got a reponse back we would pull out the relevant information from the JSON and send it to a report server which would generate a report and give us a link which we could pass on to the user. Do to issues with electricity and stuff, the pending jobs would have to be able to restart if something were to happen and the service needed to be restarted.

I’m interested in how to conceptually tackle this using Ash. I’m sure I could pretty much follow the procedure I did in Go, but it feels like Ash provides functionality that might be of use here. I’m looking at the state machine and/or possibly the reactor? I have really dug into these yet, but it “smells” like one or both of them might be useful in some way.

I’d like to know if there is some Ash-esque way of dealing with those requirements and what that would involve. The Go solution ended up working well enough, but the code was a bit of a nightmare both to maintain and to build upon.

Most Liked

sorentwo

sorentwo

Oban Core Team

You don’t need Pro to rescue stuck jobs (orphans), that’s available with the standard Lifeline plugin. The difference with Pro is it isn’t as smart about restarts.

Side note—every persistent queue may either drop executing jobs silently (like Sidekiq), or you’ll have orphans that need to be dealt with. It’s a good thing!

jimsynz

jimsynz

Ash Core Team

Be careful with this approach if you plan on ever adding distribution or any kind of rolling deploys to the application. How can it tell that it’s starting clean or it’s starting while another instance is already running?

lud

lud

If you need to be able to restart because of electricity shutdown, you must have your jobs-to-do in database. In that case a simple solution is to use Oban.

Oban will let you control how much workers you will have in your queue.

Infortunately if the server shutdowns during execution of an Oban job, on restart Oban will believe that the job is still “executing” and will not restart it (except if you use Oban Pro). But if you have a single instance of your application running, that is easy: On start, check if there is any job with the “executing” state in database and set it to “available”. And only then start the Oban queue.

Not sure how Ash could help there though.

Where Next?

Popular in Questions Top

marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
JulienCorb
I am trying to implement my new.html.eex file to create new posts on my website. new.html.eex: <h1>Create Post</h1> <...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
RisingFromAshes
I've read in another post that it may be possible with a router helper - but I couldn't find an appropriate one, and tbh, I'm still just ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New

Other popular topics Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
hariharasudhan94
Lets say i have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => "XX...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

We're in Beta

About us Mission Statement