rump13
Local queuing before invoking external URLs
Hello. I’m seeking advice on the following scenario. I have a Phoenix app that needs to invoke an external URL to put a message on an external event bus every time a significant event happens (e.g., user-created). To ensure that I don’t lose messages in case the external system is down, I would like to implement a simple local queue mechanism with durable messages or something similar in Elixir to decouple my app from the external system. Additionally, I want to avoid deploying an external system like RabbitMQ and keep it simple by running it as another process inside my Elixir app.
I have explored these options:
- Implementing the queue mechanism myself using GenStage.
- Using the Stagger library, which seems to provide exactly what I need.
- Developing a PostgreSQL producer for Broadway (although I’m new to Elixir and not yet confident in implementing this).
How would you approach something like that? Are there any other elixir native solutions that I’m missing? I would appreciate your insights and advice.
Thank you
First Post!
al2o3cr
What kind of volume are you expecting for these events?
Do you already have any kind of durable background jobs (Oban, etc) in the app?
If you do, the simplest approach would be to make “send a notification” into a job and rely on the tool.
Most Liked
v0idpwn
I implemented something similar in Kafkaesque. Since I built it for work needs, it supports only Kafka. I’m planning to add support for custom adapters in the future. Maybe it can work as a reference for you.
Other options you might consider:
- Using Oban
- OffBroadwayEcto (I’ve never used)
rump13
Thank you all for your helpful answers. After taking my enterprise integration patterns off the bookshelves and removing the dust, I discovered the outbox pattern explained there as the guaranteed delivery pattern. Then I found the following article: Implementing Message Outbox Pattern with Oban.
I believe this solution will suffice. I’ll implement Oban as a message outbox and share my experience once it’s up and running.
Popular in Questions
Other popular topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"