@samphilipd Could you highlight the differences between Rihanna and EctoJob other than the obvious dependency on Ecto?
Are there huge performance differences?
I’m trying to pick between both of them for my first job queue. (EctoJob has the advantage of changing to other databases.. cassandra etc.)
EctoJob is very much tied to Postgres.
The main difference in the design is transaction support. From what I can see, Rihanna doesn’t directly support enqueing a job as part of a larger transaction, while ecto_job strongly encourages it.
Same for job completion - both frameworks are at-least-once in the presence of crashes/restarts/etc, but ecto_job allows you to mark the job as completed in the same transaction as your other application updates.
Honeydew also recently added Ecto as a backend (honeydew/README.md at master · koudelka/honeydew · GitHub), implemented in a DB agnostic way. It will possibly hit scaling issues since it isn’t using the Postgres skip locked mechanism last I checked.
As mbuhot has explained they are intended for slightly different use-cases.
Rihanna is meant as a general purpose job queue, where your jobs might have side-effects such as POSTing to an external web service or sending an email.
ecto_job can be used for this but also allows you to execute code within the context of the job transaction, so that jobs are deleted atomically with whatever database changes you make.
Rihanna is lighter weight because it does not hold transactions open for the duration of the job execution. This allows for long-running jobs, high concurrency levels and probably higher throughput than ecto_job.
They aren’t mutually exclusive and there’s no reason you couldn’t use both in a single application if you need the atomic commit features of ecto_job as well as a general purpose job queue.
As I mentioned earlier in this thread, this isn’t true. EctoJob uses Ecto.Multi to accumulate database commands which are finally submitted to the DB in a call to Repo.transaction.
Long running network requests should be made prior to calling Repo.transaction(multi) to ensure the final transaction is not longer than necessary.
Honeydew’s Ecto Queue (as of a day or so ago), selects queries based on the database backing the Ecto Repo, in order to use DB-specific features.
The Postgres backend uses FOR UPDATE SKIP LOCKED, while the CockroachDB backend just uses ORDER BY LIMIT 1.
I’m honestly not in love with the idea of using a database as a queue, my primary motivation for writing the Ecto queue for Honeydew was so that I could transactionally enqueue work that needed to occur for every instance of a certain model.
Honeydew’s Ecto queue doesn’t even “enqueue” in the traditional sense, since it stores its state directly in the domain model, with the assumption that the job needs to be run for every instance, there’s no separate “job”.
I’ve been experimenting with a truly distributed and scalable job queue that I wrote with riak_core I’m probably going to release in a later version of Honeydew, and I think I’ll end up feeding it jobs by streaming my domain model’s database, to maintain the semantics of a transactional enqueue without the lock contention from competing workers.
How can I get a result from scheduled job from rihanna?
Rihanna.schedule(
{MyApp.Work, :need_to_get_some_result, [arg1, arg2]},
at: DateTime.from_naive!(schedule_datetime, "Etc/UTC")
)
Hi wlminimal
I’m not sure I understand what you are trying to do. What kind of result are you trying to get?
Since Rihanna is an asynchronous job processing library, it cannot return the result of the executed function. This is by design. If you need the result, why not just execute the function inline?
Regards
Sam
Hi,
Love the ability to just enqueue arbitrary calls and the shipped GUI. Really make this library easy to get up and running.
Have you considered a version that uses GitHub - lucaong/cubdb: Elixir embedded key/value database · GitHub or sqlite so you can get up and running quickly with no dependency on postgres?
Thanks,
Hafeez