Local queuing before invoking external URLs

rump13

Hello. I’m seeking advice on the following scenario. I have a Phoenix app that needs to invoke an external URL to put a message on an external event bus every time a significant event happens (e.g., user-created). To ensure that I don’t lose messages in case the external system is down, I would like to implement a simple local queue mechanism with durable messages or something similar in Elixir to decouple my app from the external system. Additionally, I want to avoid deploying an external system like RabbitMQ and keep it simple by running it as another process inside my Elixir app.

I have explored these options:

Implementing the queue mechanism myself using GenStage.
Using the Stagger library, which seems to provide exactly what I need.
Developing a PostgreSQL producer for Broadway (although I’m new to Elixir and not yet confident in implementing this).

How would you approach something like that? Are there any other elixir native solutions that I’m missing? I would appreciate your insights and advice.

Thank you

5 comments

/phoenix #queues

0 515 5

2024-04-30 03:06:41 UTC

Showing Posts 1 to 5

Show Best Posts
Show All (oldest first)
Show All (newest first)

al2o3cr

What kind of volume are you expecting for these events?

Do you already have any kind of durable background jobs (Oban, etc) in the app?

If you do, the simplest approach would be to make “send a notification” into a job and rely on the tool.

Post #1

v0idpwn

I implemented something similar in Kafkaesque. Since I built it for work needs, it supports only Kafka. I’m planning to add support for custom adapters in the future. Maybe it can work as a reference for you.

Other options you might consider:

Using Oban
OffBroadwayEcto (I’ve never used)

Post #2

rump13

The volume of messages will be low; I’m expecting something in the order of hundreds per day. I don’t have any durable background jobs for now because I’m in an early prototype stage of the app, but I’ll probably need one later. Do you think that I may want to start with something like Oban and, if necessary, replace it later if requirements change?

Something I forgot to mention in my question is that my app and the external system are running in different environments. My app is on-prem, so to reach the external event bus, I have to go to the internet.

Post #3

dimitarvp

I can’t speak how easy it would be to migrate off of Oban once you get invested in it – likely not as easy – but it otherwise covers your requirements for storing jobs somewhere and making sure each one is being executed at one point soon in the future.

And its asynchronous and not “do this absolutely immediately” nature lends itself well to the 3rd party laggy / slower APIs.

Post #4

rump13

Thank you all for your helpful answers. After taking my enterprise integration patterns off the bookshelves and removing the dust, I discovered the outbox pattern explained there as the guaranteed delivery pattern. Then I found the following article: Implementing Message Outbox Pattern with Oban.

I believe this solution will suffice. I’ll implement Oban as a message outbox and share my experience once it’s up and running.

Post #5

— All posts loaded —