tmock12

tmock12

Replacing Rails Background Jobs with an Elixir Service

Need opinions on something. One of our projects is a large Rails monolith with a ridiculous amount of background jobs. We keep creating additional jobs and adding features to them. More and more I find myself thinking “… if this was Elixir we could do this and that more efficiently…“. I’ve seen talks in the past about replacing parts of their Rails ecosystem with Elixir services.

Has anyone done this as a goal of replacing background jobs with Elixir processes? Any thoughts on that experience?

Doing a full Rails replacement would be wonderful, but is unfortunately not feasible at this point. Is introducing a new tech stack worth it? Would love any opinions from anyone that has introduced Elixir as a service to their existing apps.

Thanks!

Most Liked

hubertlepicki

hubertlepicki

Yes, that’s precisely why doing events / reactions versus background jobs is better approach IMHO. You don’t schedule a job, you send out an event somewhere. That somewhere is a queue. Then some things can pick up that event and act upon it. The major advantage of thinking this way is that you are detaching the event from implementation. If you save to database module name / function / arguments to be executed, you are also prone to the code changes - i.e. suddenly the module is not available after deploy, or it’s taking now 3 not parameters and your background queue is going to get messed up.

If you are writing your Rails app properly, i.e. there is not much logic in the models themselves, and you have stuff extracted to services, you can quite easily connect from Ecto to the same database Rails uses. It’ll understand it well, and can manipulate the same data Rails does. You obviously might need to re-implement some of the logic in Elixir but generally that’s what I would also do in Rails - background job services would be implementing own logic wherever possible to keep it isolated.

axelson

axelson

Scenic Core Team

In case anyone is still coming to this thread (based on views it appears so). Another good option for running persistent background jobs in Elixir is GitHub - oban-bg/oban: 💎 Robust job processing in Elixir, backed by modern PostgreSQL, SQLite3, and MySQL · GitHub (this is the job library that I like to use).

anthonator

anthonator

If you’re wondering if you can do this purely in Elixir, you can. However, you are going to need to roll your sleeves up and get dirty learning OTP. Personally, I think implementing a job queue/event dispatcher is a great way to learn Elixir/OTP. The use case hits on most of the OTP sweet spots and requires you to dig a little.

You’ll need to put some effort in up front to understand how advanced you want your queue to be. Specifically, how durable should your jobs be and should they be distributed? If your server crashes or restarts is it OK to lose all your jobs? Do you need/want multiple servers handling jobs? If you need durability and distribution then you’re probably better off reaching for something like Exq or Honeydew. If not then Task’s might be a good starting point.

Where Next?

Popular in Questions Top

Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
Kurisu
For example for a current url like http://localhost:4000/cosmetic/products?_utf8=✓&query=perfume&page=2, I would like to get: ...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lists...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New

Other popular topics Top

TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 41539 114
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
Lily
In templates/appointment/index.html.eex: <%= for appointment <- @appointments do %> <tr> <td><%= appoi...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 126479 1222
New

We're in Beta

About us Mission Statement