I’ve written the EventStore Elixir library which is designed for event sourcing but can also be used for event processing similar to Kafka. It uses Postgres for persistence and its built in pub/sub for triggering event notifications to subscribers.
It has support for durable and persistent (server-tracked) subscriptions guaranteeing at-least-once event delivery to subscribers. It also supports multiple subscribers to a single subscription with round-robind message distribution and partitioning if you need guaranteed ordering based upon a field within an event, such as per stream ordering but no ordering between streams.
The next EventStore release has support for stream deletion which you could use to implement Kafka style ephemeral event streams. An example might be to delete streams which are X days old or were last written to X days ago. One use of this feature is to record incoming requests as events to trigger downstream processing via one or more subcriptions. Once all the events in the stream have been processed it can be deleted.
Broadway is another library worth looking at. It is designed for building “multi-stage data ingestion and data processing pipelines”.
