giddie

giddie

Elixir-postgresql-message-queue - Pure PostgreSQL Message Queue for Elixir

Phoenix Pub/Sub is great, but I often encounter usecases where I want topic-based message passing, but durable. In other words, if the message is sent, I want to guarantee that it will eventually hit all of its configured listeners.

Oban is very popular, but the “job” abstraction feels too heavy for me. RabbitMQ+Broadway is a great combo, but it adds complexity to the deployment.

So for one project I decided to implement a reasonably flexible message queue system using only PostgreSQL. I think I’ll want to use this again, so I’ve distilled out the relevant code into a reference repo:

https://github.com/giddie/elixir-postgresql-message-queue

Usage looks a bit like this:

iex> Messaging.MessageQueueProcessor.start_link(queue: "my_queue")
iex> [%Message{type: "Example.Event", schema_version: 1, payload: %{"one" => 1}}]
...> |> Messaging.broadcast_messages!(to_queue: "my_queue")
config :postgresql_message_queue, PostgresqlMessageQueue.Messaging,
  broadcast_listeners: [
    {MyContext.MyMessageHandler, ["MyContext.Commands.*", "AnotherContext.Events.*"]},
    {MyLogger.EventLogger, ["*.Events.*"]}
  ]
@impl Messaging.MessageHandler
def handle_message(%Messaging.Message{
      type: "ExampleUsage.Events.Greeting",
      payload: %{"greeting" => greeting}
    }) do
  Logger.info("ExampleUsage: received greeting: #{greeting}")
end

I’d welcome any feedback, and I hope it’s useful to someone else, or at least interesting. I’m still unsure if this would work well as a library, but I’m thinking about it.

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Hey @giddie! What guarantees does this project provide regarding message order and visibility? That is to say, one of the common “gotchas” with Postgres is assuming that things like sequences can be relied upon to act like monotonic cursors. For example:

Process A                         | Process B
BEGIN                             | BEGIN
insert next_val('my_seq')         | insert next_val('my_seq')
1                                 | 2
# do something that takes time    | COMMIT
COMMIT

In this scenario, the value 2 will be visible outside the transaction before 1, and so if a consumer treats “oh I have seen message with id 2, therefore I am caught up to 2” they will miss messages.

How does this project avoid that issue?

giddie

giddie

I wasn’t aware of yggdrasil. Thanks :slight_smile: My main concern would be that it requires subscription. If a process relies on durable messaging, but it dies, what happens to the messages that enter the queue while the process is down? I would like guarantees that the messages will be delivered to the process once it’s up again. And it looks like that kind of guarantee may not be available for yggdrasil.

Where Next?

Popular in Announcing Top

OvermindDL1
I created a new library (rather I pulled out a couple files from my big project), it manages an operating system PID file for the BEAM. ...
New
gabrielpoca
Hello everyone! I want to share with you something that I’m really proud of: https://stillstatic.io/ Still is a static site builder for...
New
dbern
I’m excited to announce that TaxJar has developed and open-sourced DateTimeParser. We developed it because we found a need to parse user ...
New
josevalim
EDIT: since Ecto 3.0 final version is out, this post was amended to use the final versions in the instructions below. Hi everyone, We a...
New
maltoe
Hello! Came here to announce ChromicPDF, a pet project PDF generator I’ve been working on for the past few months. Why another PDF gener...
New
bluzky
You may know https://ui.shadcn.com/, a UI component library for React. I really love it’s design style and components. I’ve built some co...
384 14136 119
New
nikokozak
Hello all, I’ve been working on Svonix - a library for quickly integrating Svelte components into Phoenix views. It’s a much-needed succ...
New
RobertDober
Earmark is a pure-Elixir Markdown converter. It is intended to be used as a library (just call Earmark.as_html), but can also be used as...
239 12645 134
New
Crowdhailer
Raxx is an alternative to Plug and is inspired by projects such as Rack(Ruby) and Ring(Clojure). 1.0-rc.1 is now available. To use it re...
New
woylie
Flop is an Elixir library that applies filtering, ordering and pagination parameters to your Ecto queries. offset-based pagination with...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement