Replacing Rails Background Jobs with an Elixir Service

I/my team have done pretty much this twice, two different approaches. In first project we used RabbitMQ queues for background jobs. This was a design decision from start, so we could have workers processing those written in whatever we wanted. This is a good approach if you are already using RabbitMQ.

If you are not, but you are using Resque or Sidekiq, there might be easier path by using GitHub - akira/exq: Job processing library for Elixir - compatible with Resque / Sidekiq · GitHub - I only have limited experience with that library myself, but from what I have been told it works as advertised.

If you’re wondering if you can do this purely in Elixir, you can. However, you are going to need to roll your sleeves up and get dirty learning OTP. Personally, I think implementing a job queue/event dispatcher is a great way to learn Elixir/OTP. The use case hits on most of the OTP sweet spots and requires you to dig a little.

You’ll need to put some effort in up front to understand how advanced you want your queue to be. Specifically, how durable should your jobs be and should they be distributed? If your server crashes or restarts is it OK to lose all your jobs? Do you need/want multiple servers handling jobs? If you need durability and distribution then you’re probably better off reaching for something like Exq or Honeydew. If not then Task’s might be a good starting point.

We actually have multiple OTP apps in production and have implented different ways of trapping exits, persising state and spinning down workers for different use cases. I think that only makes it harder as we know a lot of the features and power that we would get by utilizing Elixir. So it’s not really a question of how to build it as it is a question of is it worth introducing another service as this client currently doesn’t use Elixir? I don’t have any experience using Elixir as an added service to a Rails app and am wondering if maintaining it as a service is more headache than it’s worth.

Reading back to your original question, are you running into specific problems with your Rails job implementations? You say you think you could do some things more efficiently in Elixir, but are there any actual problems on the Rails side? Is there a goal you’re trying to accomplish that moving your jobs to Elixir would help with (i.e. infrastructure costs)? Might be a premature optimization if you don’t see or aren’t expecting any issues.

Verk is another Sidekiq/Resque compatible job processing framework.

Its a larger codebase that is heavily utilized and we are at a point where we can start refactoring certain parts of the system for future maintainability. As we are creating more jobs, the codebase surrounding those jobs is continuing to grow and become more complicated. We have multi step jobs and jobs that kick off other jobs, etc. A minor subset have timed out recently so we are taking a look at some different ways to re-architect some of our jobs. We have lots of decent ruby and ruby related technology options but I wanted to explore something outside of the box and see what others experiences were like using Elixir as a service to a Rails base. A lot of our jobs are for things that could be easily handled with a language that better supported concurrency, vastly reducing the code complexity that we would have to maintain. Over-engineering and introducing dependencies (in the form of an additional language) are exactly what I’m trying to stay away from and why I asked. I just didn’t know if someone was going to tell me “yea we replaced most of our jobs with Elixir and its been low maintenance, smooth sailing and awesomeness”. Just thought I’d ask

I’m in a pretty similar situation, but as an intermediate step, I’m looking at replacing sidekiq with faktory: https://github.com/contribsys/faktory

http://contribsys.com/faktory/

faktory is made by same creator of sidekiq, but can distribute jobs to one or more servers and jobs can be executed by clients in multiple languages, including ruby and elixir. It seems like a natural progression from a rails app running jobs in something like sidekiq w/o doing a full scale rewrite of the app to elixir or elixir + phoenix. If you ever do chose to rewrite the app, then potentially could keep using existing faktory workers too.

The issue you may run into is when the job code relies on business logic defined in your active record model classes, and other service objects.

You’ll need to port that logic to elixir, and keep it in sync with the ruby version still being used from API controllers etc.

Yes, that’s precisely why doing events / reactions versus background jobs is better approach IMHO. You don’t schedule a job, you send out an event somewhere. That somewhere is a queue. Then some things can pick up that event and act upon it. The major advantage of thinking this way is that you are detaching the event from implementation. If you save to database module name / function / arguments to be executed, you are also prone to the code changes - i.e. suddenly the module is not available after deploy, or it’s taking now 3 not parameters and your background queue is going to get messed up.

If you are writing your Rails app properly, i.e. there is not much logic in the models themselves, and you have stuff extracted to services, you can quite easily connect from Ecto to the same database Rails uses. It’ll understand it well, and can manipulate the same data Rails does. You obviously might need to re-implement some of the logic in Elixir but generally that’s what I would also do in Rails - background job services would be implementing own logic wherever possible to keep it isolated.

Couldn’t agree more . In our case that’s exactly how the system is designed. There is very little logic in the Job or the Model and the jobs are essentially just actions that trigger some service to do whatever it needs to do. We would have to do some logic implementation in Elixir but looking around, its probably less than most people think.