Faktory_worker_ex - Elixir worker for Faktory

Elixir worker for Faktory.

What’s Faktory? Basically language agnostic Sidekiq (by the author of Sidekiq). I think he sums it up best with this:

Sidekiq is awesome; it’s stable and useful. It suffers from two design limitations:

  1. Sidekiq uses Redis and Redis is dumb datastore, all Sidekiq features are implemented in Ruby and have to travel over the network to access data.

  2. Because of (1), Sidekiq is limited to Ruby. You can’t execute jobs with, e.g., Python and get the same Sidekiq features.

Instead Faktory is a “smart” server that implements all major features within the server itself; the worker process can be dumb and rely on the server for job retries, reservation, Web UI, etc. This also means we can create workers in any programming language.

So anyway, here’s the Elixir worker for it: https://github.com/cjbottaro/faktory_worker_ex
It’s also on Hex.pm: https://hex.pm/packages/faktory_worker_ex

The readme is enough to get started and also lists what needs to be finished. As it is (0.1.0), it processed 10,000 jobs with concurrency of 300 without issue.

One thing not mentioned in the readme is that there is connection pooling for client connections (for enqueuing jobs), but no pooling for workers. So concurrency of 300 means 300 connections to the Faktory server. Thinking of making pooling an option… :thinking:

Anyway, I’m super excited about this and love the idea of Faktory!

7 Likes