Background job processing in Elixir

sheharyarn

I’m working on a Phoenix app that supposed to offload jobs (that scrape websites and collect data) on a background worker thread. A quick google search reveals 3 libraries:

All three use Redis at the back but Exq seems to be the most popular one, and I like the fact that it’s in Sidekiq format (can be used with the Sidekiq UI). I would like to get the community’s opinion on this. Which one are you using and why? Can someone weigh in on the Pros and Cons of each library?

Also, how hard would it be to implement a simple background job worker in elixir without relying on external libraries (or even redis for that matter)? Is that a good decision?

5 comments

/phoenix #backend

5 5488 5

2016-10-24 21:35:25 UTC

Most Liked

Show Best Posts
Show All (oldest first)
Show All (newest first)

sasajuric

Author of Elixir In Action

This is a great analysis, but I’ll nitpick on a couple of points.

Persisting the queue to disk directly from Erlang is straightforward using DETS or disk-based Mnesia table. I used the latter approach in my first system and had no problems with it.

If Redis solves it by not being distributed, so can Erlang A 3rd party non-distributed approach is not the simplest solution to a distributed problem. An in-tech (in this case pure Erlang or Elixir) solution could be simpler

I didn’t use Redis for many years, but IIRC it actually has distributed support. However, that didn’t fare well in tests made by aphyr in his Jepsen series (disclaimer: they were taken a few years ago, so maybe Redis improved since then). My takeaway from reading those was that Redis offers no strong guarantees in distributed setting. I can easily build my own in-Erlang/Elixir distributed solution that offers no real guarantees

Therefore, I don’t see compelling use case for Redis and wouldn’t recommend using it. However, some other 3rd party solution might be fine. Assuming it actually delivers on its promise, it can certainly be compelling, since unfortunately I’m not aware of currently available high-level-easy-to-use-partition-tolerant distributed abstractions in Erlang. While there are some libraries, most notably Riak Core and Riak Ensemble, the usage is not straightforward, and you need to use some forks to make them work with the most recent Erlang.

I hope this will improve with time. Phoenix Presence is a great example of a distributed abstraction built using well understood algorithms, and at the same time it seems easy to use and well documented. I hope we’ll see more of such libraries in the future.

Post #3

gregvaughn

I think you came around to the answer at the end there. IMHO it takes pretty specific requirements to make me even consider going outside beam for concurrent processing.

That said I’ve heard of verk and IIRC it also supports sidekiq format. That situation is really only useful if you have a ruby app creating background jobs to be processed in elixir.

Post #1

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Post #2

Last Post!

sasajuric

Author of Elixir In Action

I was mostly suggesting that non-distributed local-node caching is simple to do with Erlang. If you don’t need to distribute the state, then running a background job processing (which was the original problem of this thread) is as easy as starting a process from the request handler

My feeling is that people go to Redis because they used it before, everyone else uses it, and it’s seductively simple. Many, though, seem to disregard the fact that this thing has to be set up and configured somewhere, and it’s either a single point of failure (non-distributed), or otherwise unreliable.

A non-distributed in-memory cache can be as easy as ETS table, or even an Agent for smaller throughputs. Making it persistent can be easily done with DETS or non-distributed mnesia.

When it comes to managing a distributed state, the proper solution is not going to be simple with Erlang, but I don’t think distributed is ever simple or easy. Minimizing a cluster-wide state would be the first thing I’d consider. Otherwise, I’d look at libraries such as Phoenix Presence, riak_core, riak_ensemble, swarm, or syn, depending on the particular case.

Of course, reaching for 3rd party external components and databases is always a reasonable option, especially since using them can be simpler than evaluating Elixir/Erlang libraries and setting everything up properly in the code. Having a database as the single source of truth will work, but it will also be a single point of failure and a possible bottleneck. If you want to scale it, the db has to become distributed, which will then lead to similar challenges.

Post #5