How to best handle tasks when pulling from a job queue

samphilipd

I’m building a postgres-backed job queue for Elixir. The way I have it setup, I have a JobDispatcher that polls the DB every N milliseconds for available jobs. It queries the number of jobs that it has available capacity for and then distributes these to worker processes.

I can imagine two ways to implement this:

One would be to have a separate WorkerSupervisor that spins up a certain number of Worker GenServers (let’s say 25). Each Worker registers itself with the JobDispatcher to say that it is available to receive jobs. Then the JobDispatcher sends the job via message to the existing GenServer and monitors it for either a “job completed” message (meaning it became available for work again) or an exit (if it crashed).

The other way is to not worry about keeping the worker processes running at all, and instead to have the JobDispatcher spawn a one-shot tasks for each new job (using Task.Supervisor.async_nolink) which it keeps a monitor on, and handles both normal and dirty exits. This would appear to have the advantage of simplicity, plus garbage collection is kept localized to each Task so it might also be more efficient.

I haven’t worked enough with OTP patterns to know which one of these is the better approach, does anybody have any ideas?

33 comments

#task #processes #genserver

3 9415 33

2023-02-22 11:05:38 UTC

Showing Posts 33 to 24

Show Best Posts
Show All (oldest first)
Show All (newest first)

outlog

latest Oban supports sqlite oban | Hex

Post #34

macrocreation

True. It does now and it’s cool. I ended up going with ExJob - fits with what I want to do.

Post #33

LostKobrakai

CubDB barely existed at the time of the post you answered

Post #32

macrocreation

Hi,

Love the ability to just enqueue arbitrary calls and the shipped GUI. Really make this library easy to get up and running.

Have you considered a version that uses GitHub - lucaong/cubdb: Elixir embedded key/value database · GitHub or sqlite so you can get up and running quickly with no dependency on postgres?

Thanks,
Hafeez

Post #31

samphilipd

Hi wlminimal

I’m not sure I understand what you are trying to do. What kind of result are you trying to get?

Since Rihanna is an asynchronous job processing library, it cannot return the result of the executed function. This is by design. If you need the result, why not just execute the function inline?

Regards
Sam

Post #30

wlminimal

How can I get a result from scheduled job from rihanna?

Rihanna.schedule(
    {MyApp.Work, :need_to_get_some_result, [arg1, arg2]}, 
    at: DateTime.from_naive!(schedule_datetime, "Etc/UTC")
)

Post #29

koudelka

Honeydew’s Ecto Queue (as of a day or so ago), selects queries based on the database backing the Ecto Repo, in order to use DB-specific features.

The Postgres backend uses FOR UPDATE SKIP LOCKED, while the CockroachDB backend just uses ORDER BY LIMIT 1.

I’m honestly not in love with the idea of using a database as a queue, my primary motivation for writing the Ecto queue for Honeydew was so that I could transactionally enqueue work that needed to occur for every instance of a certain model.

Honeydew’s Ecto queue doesn’t even “enqueue” in the traditional sense, since it stores its state directly in the domain model, with the assumption that the job needs to be run for every instance, there’s no separate “job”.

I’ve been experimenting with a truly distributed and scalable job queue that I wrote with riak_core I’m probably going to release in a later version of Honeydew, and I think I’ll end up feeding it jobs by streaming my domain model’s database, to maintain the semantics of a transactional enqueue without the lock contention from competing workers.

Post #28

mbuhot

As I mentioned earlier in this thread, this isn’t true. EctoJob uses Ecto.Multi to accumulate database commands which are finally submitted to the DB in a call to Repo.transaction.

Long running network requests should be made prior to calling Repo.transaction(multi) to ensure the final transaction is not longer than necessary.

Post #27

namor

Thanks mbuhot & samphilipd. I have a much better idea now of what to do next

Post #26

samphilipd

As mbuhot has explained they are intended for slightly different use-cases.

Rihanna is meant as a general purpose job queue, where your jobs might have side-effects such as POSTing to an external web service or sending an email.

ecto_job can be used for this but also allows you to execute code within the context of the job transaction, so that jobs are deleted atomically with whatever database changes you make.

Rihanna is lighter weight because it does not hold transactions open for the duration of the job execution. This allows for long-running jobs, high concurrency levels and probably higher throughput than ecto_job.

They aren’t mutually exclusive and there’s no reason you couldn’t use both in a single application if you need the atomic commit features of ecto_job as well as a general purpose job queue.

Post #25

Load more posts (23 remaining)