sheharyarn

sheharyarn

A few months back I created this thread asking everyone’s opinion on different background job processing libraries available for Elixir and was encouraged to go with a simple GenServer at the time.

I eventually needed to persist job state and was reminded of this table from @sasajuric’s Elixir in Action (and his tweet):

So I decided to stay in BEAM-land and used Mnesia in my project for a while before eventually releasing it as (yet another) background job processing library.

I would really appreciate if I could get the community’s feedback on the project implementation and some pointers, especially on:

  • The way child processes are currently handled
  • Testing the Supervisor and GenServer
    • What’s the right way of testing something like this?
  • Adding Delayed Jobs
    • Should I create another GenServer for Delayed Jobs or modify the current one?

Thank you! :smile:


You can check out the project on Github:

Que

Showing Posts 1 to 9

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

This definitely seems interesting. Just a few comments:

  1. It seems like the workers are configured under the Que application and not the user application. This can be an issue if the tasks use the user’s database for example.

Que starts, grabs a persisted job, and tries to run it. If the job talks to the database immediately the user’s Repo process may not yet be up.

  1. Unfortunately your Queue module isn’t a queue, it’s a stack. que/lib/que/queue.ex at master · sheharyarn/que · GitHub. You’re placing new jobs on the front of a list, and then pulling jobs off also from the front of a list, treating it as a stack not a queue.

  2. There seems to be no real isolation between queues. This is problematic for a variety of reasons. From a performance perspective everything is serialized through your Que.Server genserver, which won’t scale as you have more queues or as particular queues become very busy.

It’s also an issue from a fault tolerance perspective. Any issue with one queue will nuke every other queue.

  1. Isolation between workers is limited. https://github.com/sheharyarn/que/blob/master/lib/que/queue_set.ex#L102 goes through each queue completely synchronously and then just runs que/lib/que/queue.ex at master · sheharyarn/que · GitHub some N jobs in each queue under the main single task supervisor.

Each job is executed in its own process which is good, but by putting them under all the same supervisor with the same settings all you need is 5 job failures in 3 seconds to take down all the other jobs. You can of course just adjust the supervisor settings but even still, different supervisors for different queues would be better.

The root issue is that you do 99% of the stuff with a single genserver. There also appear to be race conditions. For example, two nodes starting at the same time will both query mnesia for incomplete jobs. You do that in a transaction, so one will run after another, but since you don’t update the state in that transaction they’re both still gonna get the same list of incomplete jobs and both try to run them.

I apologize that all of this is so negative. The issue is that while Erlang and Elixir are indeed good fits for the items on that list you have, the reason that they’re good is because Erlang and Elixir offer excellent primitives with which to solve the problem. Those primitives still need to be used correctly, and doing so can actually be pretty hard because the problems themselves are pretty hard.

brightball

brightball

Only comment I’d have here is that there is a Ruby library called Que that uses Postgres advisory locks for some speed perks and has gained in popularity a good bit.

Just to avoid naming conflicts it might be a good idea to change the name to something more distinctive? I’d heard that there was potentially going to be an Elixir port of that library and if that happens you’re going to get a lot more confusion. If it was a library from any other language I wouldn’t be that concerned.

sheharyarn

sheharyarn OP

How would I go about implementing it like that?

It is a queue. I’m placing jobs at the end and pulling them from the front. I guess the push and pop method names here are misleading. I should change them to something like in and out.

This is an interesting issue. I don’t know much about OTP to understand how to approach this. Should I create multiple Supervisors under one application, one supervisor supervising multiple supervisors, each for one worker or one supervisor with multiple GenServers each for one Worker? Either way, how can I automatically start the supervisors / genservers for each defined worker? Any tips on doing this the right way?

Not at all. I really appreciate you taking the time out to go through the code and give feedback. My goal is to improve my Elixir and OTP skills. :smiley:

sheharyarn

sheharyarn OP

I didn’t know about the Ruby Que library. In fact, I “cleverly” came up with the name by replacing K with Q in the Node.js Kue library.

Also, I don’t think there’s a way to change package names once they’ve been published to Hex. :frowning:

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

This is true, my apologies. Unfortunately however its implementation is such that adding N jobs distinctly produces N^2 work. If you do:

for image <- images do
  Que.add(App.Workers.ImageConverter, some_image)
end

you’re going to do N^2 work, when only linear is necessary if using something like :queue, or following its internal implementaiton. Doing ++ to the end of a growing list is generally not what you want to do.

You provide an API like poolboy for example, where you have a child_spec function and then people place it in their supervision tree in their own application.

There’s a lot to say here, I’ll have to reply tomorrow.

OvermindDL1

OvermindDL1

This is the use case for a zipper. :slight_smile:

sheharyarn

sheharyarn OP

Thank you for being patient with me and taking the time out to answer my questions. I went back, did some more research and started refactoring the application trying to implement your suggestions (the OTP-related ones for now):


Before I try to do the same for TaskSupervisor, a few questions:

  • Does this application structure make sense?

  • For now I’m using {:global, {Que.Server, SomeWorker}} as the GenServer names since it’s simple, but according to this post, using :global introduces extra overhead.

    • Should I keep using it?
    • Roll my own ServerRegistry GenServer like in this tutorial or go with something like gproc ?
    • Use Elixir’s own Registry?
    • Or take the simplest approach and use the worker module names as the server names?
  • When a new job is added, the ServerSupervisor checks if a GenServer for that worker already exists. If not, it spawns one. Is this the right approach?

  • Should I also keep the same structure for TaskSupervisor?

    • i.e. TaskSupervisor > TaskSupervisor.SomeWorker > Tasks

Thanks again.

Qqwy

Qqwy

TypeCheck Core Team

I’d say a zipper is overkill here, as we are only ever appending at the end and reading from the front. You just need one of Okasaki’s purely functional queue algorithms, of which :queue and multiple Elixir wrappers on Hex such as e_queue are implementations.

OvermindDL1

OvermindDL1

True true, just of the aspect of doing it manually. And it is not like zippers are difficult, just keep a tuple of two elements, pull off the front of one, push to the front of the other, when the pull one is empty then swap them and reverse, it is quite efficient and simple. :slight_smile:

— All posts loaded —

Where Next? Top

Trending in Announcing Top

achempion
Hi, I would like to tell about my initiative to further maintain and develop Waffle project which is the fork of Arc library. The progre...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New
charlenopires
Hi everyone :waving_hand: I’ve been working on PhiaUI, an open-source UI component library for Phoenix LiveView, and I wanted to share i...
New
gtcode
Aloha gang, There’s a new class of AI model out from TypeSafe. Jev is their flagship model. You can learn about it here: ...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
raa-ziq
Hi everyone :waving_hand: I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
AstonJ
With advancements in AI gathering pace and set to increasingly accelerate, perhaps it’s worth having an annual thread on the topic - feel...
New
bartblast
If you haven’t tried Hologram yet, what’s stopped you? Here’s why I’m asking. Building a framework turns out to be two jobs, the framewo...
New
mat-hek
Popcorn, the library that runs Elixir code in the browser, is now stable :popcorn: This is possible because it now builds on a very stab...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews