Looking for a relaible event bus library

Questions / Help
1

Hi,

I’m looking for an event bus library that reliably implements Outbox pattern, i.e. can guarantee that if the publishing function finished its execution, then a message will be delivered at least once to every subscriber.

That includes postponing parent shutdown until getting an acknowledgement from the storage that an event was successfully persisted.

Is there anything like that in the Elixir land?

1 Like
2

maybe just MQTT? QoS=2, retained messages.

3

@michalmuskala what do you think about sticking GitHub - michalmuskala/persistent_ets into combining Search · ets.new · GitHub in the context of the question above?

How irresponsible does that looks like to you :slight_smile: ?

4

Well, telemetry guarantees that every subscriber function will be called exactly once.

2 Likes
5

Loosing a telemetry event is not at the same cost as loosing domain event.

Event bus should persist event before returning ‘ok’, so that it will survive node shutdown .

6

Are the messages to be published to an external service or inside the same Elixir system?

For the Outbox pattern we use Oban as it has retry built-in. If you are publishing to services like Kafka or RabbitMQ you still have to trust the service to deliver the message if the publishing function returns.

7

Interesting! How are you using Oban for a message delivery?

I’d prefer to have an internal service, but not sure if it’s possible to achieve an at-least-once delivery whilst running event bus on the same node.

8

While in a DB transaction that creates some stuff we insert an Oban job (outbox pattern itself).

When the transaction succeeds we can return “OK” from our API and guarantee that the job will be performed at some point since the job was inserted in Postgres.

In the job there is a simple HTTP call to another service or a message published to Kafka. It can be anything.

1 Like
9

that’s neat! thanks for sharing :slight_smile:

10

That might be an overkill for you, but Debezium, so a Change Data Capture solution usually used with Kafka, can be a great fit for that pattern.

It works more or less like this:

  • your outbox is a table in your DB, so you write to it with full ACID guarantees,
  • Debezium pretends to be a read replica for your DB and observes its transaction log,
  • for every change in your DB, Debezium emits an appropriate event to a Kafka topic,
  • you can consume those events in other apps.
1 Like