giddie

giddie

Elixir-postgresql-message-queue - Pure PostgreSQL Message Queue for Elixir

Phoenix Pub/Sub is great, but I often encounter usecases where I want topic-based message passing, but durable. In other words, if the message is sent, I want to guarantee that it will eventually hit all of its configured listeners.

Oban is very popular, but the “job” abstraction feels too heavy for me. RabbitMQ+Broadway is a great combo, but it adds complexity to the deployment.

So for one project I decided to implement a reasonably flexible message queue system using only PostgreSQL. I think I’ll want to use this again, so I’ve distilled out the relevant code into a reference repo:

https://github.com/giddie/elixir-postgresql-message-queue

Usage looks a bit like this:

iex> Messaging.MessageQueueProcessor.start_link(queue: "my_queue")
iex> [%Message{type: "Example.Event", schema_version: 1, payload: %{"one" => 1}}]
...> |> Messaging.broadcast_messages!(to_queue: "my_queue")
config :postgresql_message_queue, PostgresqlMessageQueue.Messaging,
  broadcast_listeners: [
    {MyContext.MyMessageHandler, ["MyContext.Commands.*", "AnotherContext.Events.*"]},
    {MyLogger.EventLogger, ["*.Events.*"]}
  ]
@impl Messaging.MessageHandler
def handle_message(%Messaging.Message{
      type: "ExampleUsage.Events.Greeting",
      payload: %{"greeting" => greeting}
    }) do
  Logger.info("ExampleUsage: received greeting: #{greeting}")
end

I’d welcome any feedback, and I hope it’s useful to someone else, or at least interesting. I’m still unsure if this would work well as a library, but I’m thinking about it.

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Hey @giddie! What guarantees does this project provide regarding message order and visibility? That is to say, one of the common “gotchas” with Postgres is assuming that things like sequences can be relied upon to act like monotonic cursors. For example:

Process A                         | Process B
BEGIN                             | BEGIN
insert next_val('my_seq')         | insert next_val('my_seq')
1                                 | 2
# do something that takes time    | COMMIT
COMMIT

In this scenario, the value 2 will be visible outside the transaction before 1, and so if a consumer treats “oh I have seen message with id 2, therefore I am caught up to 2” they will miss messages.

How does this project avoid that issue?

giddie

giddie

I wasn’t aware of yggdrasil. Thanks :slight_smile: My main concern would be that it requires subscription. If a process relies on durable messaging, but it dies, what happens to the messages that enter the queue while the process is down? I would like guarantees that the messages will be delivered to the process once it’s up again. And it looks like that kind of guarantee may not be available for yggdrasil.

Last Post!

giddie

giddie

Well, that’s kind of my point with this project – different usecases will have different requirements. So a library that tries to cover all the usecases is not necessarily a good idea. But some reference code can be really handy, so you can copy the bits you need and adapt it as needed.

Well no, it’s a data stream processing framework. So for processing a message queue, it’s pretty useful. It’s certainly not the only way to ingest a queue, though. A simpler GenServer for queue processing can be found here in my CQRS patterns repo, and it’s pretty much a drop-in replacement for the Broadway-based processor.

Ultimately, anything could call Messaging.process_message_queue_batch/2, so the world’s your oyster.

The main advantage of Broadway in this message queue example is concurrency – it decouples the ingestion from the processing, so one process can pull a batch of 100 messages, and distribute them across a pool of 5 workers. And of course if the work calls for it, Broadway has features such as batching and pre-fetching that could be applied very easily on top of this, since the hard work of building the Broadway Producer is already done,

Where Next?

Popular in Announcing Top

KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36820 110
New
Crowdhailer
Raxx is an alternative to Plug and is inspired by projects such as Rack(Ruby) and Ring(Clojure). 1.0-rc.1 is now available. To use it re...
New
pkrawat1
Hey guyz We at @aviabird are working on a payment library in elixir/phoenix. We are targeting March 2018 to add 56 Gateways to it. Have...
New
oltarasenko
Dear Elixir community, After a year of development, bug fixes, and improvements, we are proudly ready to share the release of Crawly 0.1...
New
fuelen
Hey folks! Want to present a toolkit for writing command-line user interfaces. It provides a convenient interface for colorizing text...
New
tfwright
After working on it for a couple of months and using it in production for most of that time, today I’ve released LiveAdmin, a LiveView ba...
New
zachdaniel
Ash Framework What is Ash? Ash Framework is a declarative, resource-oriented application development framework for Elixir. A resource can...
New

Other popular topics Top

jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 44265 214
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New

We're in Beta

About us Mission Statement