koudelka
I’ve just released version 1.1.1 of the Honeydew job queue library and added “instant” Ecto work queues to turn your schema into a job queue.
By adding a couple fields to your schema, Honeydew will automatically run background jobs related to your schema on your worker nodes whenever you insert a new row. For instance, if a user uploads a song file, and you want to transcode it into a number of different formats, or if a user uploads a photo and you want to run your object classifier against it.
This is done without a separate database table, and entirely without an external stateful queue process for you to worry about. Since your schema’s table is the work queue itself, they’re always in sync and issues relating to the the lack of atomicity between your row’s insertion and the enqueue are eliminated, in fact, your application code never enqueues jobs at all.
This specific queue type integrates well into the rest of the Honeydew ecosystem, supporting almost all of the library’s features (failure modes, suspend/resume, etc).
Check out the README for a lengthier description and an example project.
Hope folks find it useful, feedback is always welcome. 
Where Next?
Trending in Announcing
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #phoenix_html
- #iex
- #graphql
- #ai
- #genstage
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 2- Show Best Posts
- Show All Posts (oldest first)
- Show All Posts (newest first)
mbuhot
Awesome! Another option for transactionally consistent durable queueing
Have you tested the throughput? I’ve read that without DB specific tricks like PostgreSQL
skip lockedthat simpleupdate order by limit 1queries can be a bottleneck.koudelka
Thanks for the heads up!
I haven’t run any benchmarks yet, outside of hammering the queue in an attempt to flush out any flagrant concurrency issues. My first goal is to get it correct, and then optimize for speed.
I’m using it with CockroachDB, which doesn’t support
skip locked, unfortunately. But I don’t think it’d be a big deal to change out which SQL statements are used based on the database (either trying to auto-detect at runtime, or requiring the user to indicate).