Replacing Rails Background Jobs with an Elixir Service

[tmock12]

Need opinions on something. One of our projects is a large Rails monolith with a ridiculous amount of background jobs. We keep creating additional jobs and adding features to them. More and more I find myself thinking “… if this was Elixir we could do this and that more efficiently…“. I’ve seen talks in the past about replacing parts of their Rails ecosystem with Elixir services.

Has anyone done this as a goal of replacing background jobs with Elixir processes? Any thoughts on that experience?

Doing a full Rails replacement would be wonderful, but is unfortunately not feasible at this point. Is introducing a new tech stack worth it? Would love any opinions from anyone that has introduced Elixir as a service to their existing apps.

Thanks!

[hubertlepicki]

I/my team have done pretty much this twice, two different approaches. In first project we used RabbitMQ queues for background jobs. This was a design decision from start, so we could have workers processing those written in whatever we wanted. This is a good approach if you are already using RabbitMQ.

If you are not, but you are using Resque or Sidekiq, there might be easier path by using https://github.com/akira/exq - I only have limited experience with that library myself, but from what I have been told it works as advertised.

[anthonator]

If you’re wondering if you can do this purely in Elixir, you can. However, you are going to need to roll your sleeves up and get dirty learning OTP. Personally, I think implementing a job queue/event dispatcher is a great way to learn Elixir/OTP. The use case hits on most of the OTP sweet spots and requires you to dig a little.

You’ll need to put some effort in up front to understand how advanced you want your queue to be. Specifically, how durable should your jobs be and should they be distributed? If your server crashes or restarts is it OK to lose all your jobs? Do you need/want multiple servers handling jobs? If you need durability and distribution then you’re probably better off reaching for something like Exq or Honeydew. If not then Task’s might be a good starting point.

[tmock12]

We actually have multiple OTP apps in production and have implented different ways of trapping exits, persising state and spinning down workers for different use cases. I think that only makes it harder as we know a lot of the features and power that we would get by utilizing Elixir. So it’s not really a question of how to build it as it is a question of is it worth introducing another service as this client currently doesn’t use Elixir? I don’t have any experience using Elixir as an added service to a Rails app and am wondering if maintaining it as a service is more headache than it’s worth.

[anthonator]

Reading back to your original question, are you running into specific problems with your Rails job implementations? You say you think you could do some things more efficiently in Elixir, but are there any actual problems on the Rails side? Is there a goal you’re trying to accomplish that moving your jobs to Elixir would help with (i.e. infrastructure costs)? Might be a premature optimization if you don’t see or aren’t expecting any issues.

[axelson]

Verk is another Sidekiq/Resque compatible job processing framework.

[tmock12]

Its a larger codebase that is heavily utilized and we are at a point where we can start refactoring certain parts of the system for future maintainability. As we are creating more jobs, the codebase surrounding those jobs is continuing to grow and become more complicated. We have multi step jobs and jobs that kick off other jobs, etc. A minor subset have timed out recently so we are taking a look at some different ways to re-architect some of our jobs. We have lots of decent ruby and ruby related technology options but I wanted to explore something outside of the box and see what others experiences were like using Elixir as a service to a Rails base. A lot of our jobs are for things that could be easily handled with a language that better supported concurrency, vastly reducing the code complexity that we would have to maintain. Over-engineering and introducing dependencies (in the form of an additional language) are exactly what I’m trying to stay away from and why I asked. I just didn’t know if someone was going to tell me “yea we replaced most of our jobs with Elixir and its been low maintenance, smooth sailing and awesomeness”. Just thought I’d ask

[beaorn]

I’m in a pretty similar situation, but as an intermediate step, I’m looking at replacing sidekiq with faktory: https://github.com/contribsys/faktory

http://contribsys.com/faktory/

faktory is made by same creator of sidekiq, but can distribute jobs to one or more servers and jobs can be executed by clients in multiple languages, including ruby and elixir. It seems like a natural progression from a rails app running jobs in something like sidekiq w/o doing a full scale rewrite of the app to elixir or elixir + phoenix. If you ever do chose to rewrite the app, then potentially could keep using existing faktory workers too.

[mbuhot]

The issue you may run into is when the job code relies on business logic defined in your active record model classes, and other service objects.

You’ll need to port that logic to elixir, and keep it in sync with the ruby version still being used from API controllers etc.

[hubertlepicki]

The issue you may run into is when the job code relies on business logic defined in your active record model classes, and other service objects.

Yes, that’s precisely why doing events / reactions versus background jobs is better approach IMHO. You don’t schedule a job, you send out an event somewhere. That somewhere is a queue. Then some things can pick up that event and act upon it. The major advantage of thinking this way is that you are detaching the event from implementation. If you save to database module name / function / arguments to be executed, you are also prone to the code changes - i.e. suddenly the module is not available after deploy, or it’s taking now 3 not parameters and your background queue is going to get messed up.

If you are writing your Rails app properly, i.e. there is not much logic in the models themselves, and you have stuff extracted to services, you can quite easily connect from Ecto to the same database Rails uses. It’ll understand it well, and can manipulate the same data Rails does. You obviously might need to re-implement some of the logic in Elixir but generally that’s what I would also do in Rails - background job services would be implementing own logic wherever possible to keep it isolated.

[tmock12]

If you are writing your Rails app properly, i.e. there is not much logic in the models themselves, and you have stuff extracted to services, you can quite easily connect from Ecto to the same database Rails uses. It’ll understand it well, and can manipulate the same data Rails does. You obviously might need to re-implement some of the logic in Elixir but generally that’s what I would also do in Rails - background job services would be implementing own logic wherever possible to keep it isolated.

Couldn’t agree more . In our case that’s exactly how the system is designed. There is very little logic in the Job or the Model and the jobs are essentially just actions that trigger some service to do whatever it needs to do. We would have to do some logic implementation in Elixir but looking around, its probably less than most people think.

[dogweather]

I’ve been implementing every new API point that doesn’t use ActiveRecord business logic as an HTTP service in Elixir. It’s been working out great - seeing lots of advantages: easier development, fast, low memory.

[dwahyudi]

The easiest way to this is to make an independent Elixir (web)app. Rewrite some of your Rails app code with Elixir code and let the Rails app communicate with that Elixir app via rest HTTP.

If necessary, let the Elixir app access the same database and probably share the users session (if any) with the Rails app.

The downside is of course on the added latency when communicating between these two.

[eidge]

We’ve been using a similar strategy for our push notification needs.

The basic architecture is Rails enqueues jobs in Redis (using Resque), but these jobs never get consumed by Resque. Instead an Elixir app running Exq consumes and runs these jobs.

It’s been pretty much smooth sailing consuming up to thousands of jobs per second. It is however extra infrastructure complexity (one more machine to provision, run and monitor along with a different technology stack to maintain, and learn how to run in production). In our case this complexity was well justified by the simplicity of the implementation and the load it withstands - 2 weeks worth of implementation.

If we were using Ruby, we would need a lot of processes (and therefore memory) to be able to run at the same level of concurrency.

Your mileage might vary though.

[cjbottaro]

Verk is another Sidekiq/Resque compatible job processing framework.

I don’t understand why people keep pushing things like this when the author of Sidekiq himself has created a language agnostic Sidekiq called Faktory. Where the mission statement is practically, “You know how ubiquitous and awesome Sidekiq is? Now it’s available in any language.”

I’m in a pretty similar situation, but as an intermediate step, I’m looking at replacing sidekiq with faktory: https://github.com/contribsys/faktory

http://contribsys.com/faktory/

faktory is made by same creator of sidekiq, but can distribute jobs to one or more servers and jobs can be executed by clients in multiple languages, including ruby and elixir. It seems like a natural progression from a rails app running jobs in something like sidekiq w/o doing a full scale rewrite of the app to elixir or elixir + phoenix. If you ever do chose to rewrite the app, then potentially could keep using existing faktory workers too.

contribsys/faktory

Just sayin…

Look, I’m an Elixir zealot at this point, and I want the language to succeed. And I think normalizing around something like Faktory as the defacto standard can help with that.

[cjbottaro]

If it wasn’t immediately obvious from the previous post, the official Faktory wiki lists an Elixir client.

Full disclaimer, I’m the main author of the Elixir client and I also worked with Mike Perham years ago.

[axelson]

Well the (upcoming) pricing

“Advanced features for the advanced business, Price TBD.”

Is a reason to be wary. I know some of the features under Sidekiq Pro/Enterprise are features that could be considered to be important to a large number of use cases (such as rate limiting and periodic jobs), but generally I’m just trying to provide insight into the multiple available options.

[cjbottaro]

If you don’t want to write/manage these features… someone else will do it for you… for a price.

My company used Sidekiq Pro for about 3 years, then abandoned it for OS Sidekiq with our own, in house developed middleware and extensions.

What’s there to be wary about? It’s literally open source.

[beaorn]

I think to have robust features with a level of support business critical applications require, the model that Mike Perham has taken with sidekiq and going to take with faktory is ideal. You get great features for free that are open source that people can build solutions on top of, BUT if they dont exist out in the open source space or the ones that do arent maintained, you can get enterprise level features through the paid levels they provide. I also think that lets him (and any team) make a living keeping the open source implementations maintained alive and usable that tons of apps are using for free. I’m pumped about faktory 1.0

[axelson]

In case anyone is still coming to this thread (based on views it appears so). Another good option for running persistent background jobs in Elixir is https://github.com/sorentwo/oban/ (this is the job library that I like to use).