How to best handle tasks when pulling from a job queue

[Qqwy]

Wonderful!

@samphilipd

@jordiee

If something goes wrong in pulling and the genserver crashes you also lose all currently being executed jobs correct?

Jobs are spawned underneath a Task.Supervisor so they will run to completion regardless of the state of the Dispatcher that spawned them. Since the Dispatcher handles marking jobs as failed/completed, these jobs will never register their status with the database so the lock will be lost and the respawned dispatcher will try them over again.

The result will be that some jobs are run twice.

A great way to handle this edge case. Writing jobs to ensure that they will result in a no-op when ran again if required is usually not hard, and a great solution.

[namor]

@samphilipd Could you highlight the differences between Rihanna and EctoJob other than the obvious dependency on Ecto?

Are there huge performance differences?

I’m trying to pick between both of them for my first job queue. (EctoJob has the advantage of changing to other databases… cassandra etc.)

[mbuhot]

EctoJob is very much tied to Postgres.
The main difference in the design is transaction support. From what I can see, Rihanna doesn’t directly support enqueing a job as part of a larger transaction, while ecto_job strongly encourages it.
Same for job completion - both frameworks are at-least-once in the presence of crashes/restarts/etc, but ecto_job allows you to mark the job as completed in the same transaction as your other application updates.

[mbuhot]

Honeydew also recently added Ecto as a backend (https://github.com/koudelka/honeydew/blob/master/README.md#ecto-poll-queue), implemented in a DB agnostic way. It will possibly hit scaling issues since it isn’t using the Postgres skip locked mechanism last I checked.

[samphilipd]

@namor

@samphilipd Could you highlight the differences between Rihanna and EctoJob other than the obvious dependency on Ecto?

Are there huge performance differences?

I’m trying to pick between both of them for my first job queue. (EctoJob has the advantage of changing to other databases… cassandra etc.)

As mbuhot has explained they are intended for slightly different use-cases.

Rihanna is meant as a general purpose job queue, where your jobs might have side-effects such as POSTing to an external web service or sending an email.

ecto_job can be used for this but also allows you to execute code within the context of the job transaction, so that jobs are deleted atomically with whatever database changes you make.

Rihanna is lighter weight because it does not hold transactions open for the duration of the job execution. This allows for long-running jobs, high concurrency levels and probably higher throughput than ecto_job.

They aren’t mutually exclusive and there’s no reason you couldn’t use both in a single application if you need the atomic commit features of ecto_job as well as a general purpose job queue.

[namor]

Thanks mbuhot & samphilipd. I have a much better idea now of what to do next

[mbuhot]

@samphilipd

Rihanna is lighter weight because it does not hold transactions open for the duration of the job execution

As I mentioned earlier in this thread, this isn’t true. EctoJob uses Ecto.Multi to accumulate database commands which are finally submitted to the DB in a call to Repo.transaction.

Long running network requests should be made prior to calling Repo.transaction(multi) to ensure the final transaction is not longer than necessary.

[koudelka]

Honeydew’s Ecto Queue (as of a day or so ago), selects queries based on the database backing the Ecto Repo, in order to use DB-specific features.

The Postgres backend uses FOR UPDATE SKIP LOCKED, while the CockroachDB backend just uses ORDER BY LIMIT 1.

I’m honestly not in love with the idea of using a database as a queue, my primary motivation for writing the Ecto queue for Honeydew was so that I could transactionally enqueue work that needed to occur for every instance of a certain model.

Honeydew’s Ecto queue doesn’t even “enqueue” in the traditional sense, since it stores its state directly in the domain model, with the assumption that the job needs to be run for every instance, there’s no separate “job”.

I’ve been experimenting with a truly distributed and scalable job queue that I wrote with riak_core I’m probably going to release in a later version of Honeydew, and I think I’ll end up feeding it jobs by streaming my domain model’s database, to maintain the semantics of a transactional enqueue without the lock contention from competing workers.

[wlminimal]

How can I get a result from scheduled job from rihanna?

Rihanna.schedule(
    {MyApp.Work, :need_to_get_some_result, [arg1, arg2]}, 
    at: DateTime.from_naive!(schedule_datetime, "Etc/UTC")
)

[samphilipd]

Hi wlminimal

I’m not sure I understand what you are trying to do. What kind of result are you trying to get?

Since Rihanna is an asynchronous job processing library, it cannot return the result of the executed function. This is by design. If you need the result, why not just execute the function inline?

Regards
Sam

[macrocreation]

Hi,

Love the ability to just enqueue arbitrary calls and the shipped GUI. Really make this library easy to get up and running.

Have you considered a version that uses GitHub - lucaong/cubdb: Elixir embedded key/value database or sqlite so you can get up and running quickly with no dependency on postgres?

Thanks,
Hafeez

[LostKobrakai]

CubDB barely existed at the time of the post you answered

[macrocreation]

True. It does now and it’s cool. I ended up going with ExJob - fits with what I want to do.

[outlog]

@macrocreation

or sqlite

latest Oban supports sqlite oban | Hex