What would it take to create Kafka like solution in Elixir?

I’ve written the EventStore Elixir library which is designed for event sourcing but can also be used for event processing similar to Kafka. It uses Postgres for persistence and its built in pub/sub for triggering event notifications to subscribers.

It has support for durable and persistent (server-tracked) subscriptions guaranteeing at-least-once event delivery to subscribers. It also supports multiple subscribers to a single subscription with round-robind message distribution and partitioning if you need guaranteed ordering based upon a field within an event, such as per stream ordering but no ordering between streams.

The next EventStore release has support for stream deletion which you could use to implement Kafka style ephemeral event streams. An example might be to delete streams which are X days old or were last written to X days ago. One use of this feature is to record incoming requests as events to trigger downstream processing via one or more subcriptions. Once all the events in the stream have been processed it can be deleted.

Broadway is another library worth looking at. It is designed for building “multi-stage data ingestion and data processing pipelines”.

6 Likes

Question for those of you that know Kafka better than I do.
Doesn’t :disk_log supply most of the “primitives” needed to implement a simple local/single node Kafka-like utility?

1 Like

I experimented with disk_log as a Kafka-like event store for a while. Ultimately I decided to go a different route but the discussion might be interesting if it’s something you’re looking at tackling.

3 Likes

Completely forgot about this, but there was a nice talk about an opinionated kafka library made by someone from Square Enix (if I’m not mistaken).
https://codesync.global/media/from-legacy-to-event-sourcing/

3 Likes