onkara
Just went through this great discussion thread on Big Data with Elixir and looking at this diagram I couldn’t help but ask can Broadway be a replacement for Kafka ? But on further research it turns out Broadway is a way of creating efficient data processing pipelines. And as such it sits more downstream on the consumer side of the equation. Though one can see a consumer further producing something which then is picked up by another data processing pipelines.
There have been attempts like ErlBus, EventBus and Phoenix’s own PubSub which do what Kafka does (except fault tolerant persistence among others). ErlBus supports distributed PubSub architecture.
The above being the background/context. My question is why do the various message bus stories in BEAM land don’t even come close to Kafka and what would it take to create Kafka competitor in Elixir? Is there some limitation in BEAM that does not easily lend itself to Kafkaesque architecture?
Would love to hear your thoughts/insights?
Where Next?
Trending in Discussions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Bringing learning, work and compliance together since 1987.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync - join us in 2026!
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
WFransen
Hello, I’m not sure if it is useful but maybe this post can offer some insights as well:
https://forum.elixirforum.com/t/can-we-beat-kafka-if-we-build-it-in-elixir
dimitarvp
Probably nobody thinks it’s worth duplicating effort. Erlang’s BEAM is quite well suited for append-only logs like Kafka’s otherwise.
sribe
RabbitMQ
onkara
RabbitMQ is just a messaging queue. There is difference between the two and they cater to different use cases
madshargreave
I’ve played around with prototypes for Kafka like messaging systems in Elixir. It’s definitely possible and I believe it to be a simpler implementation than its JVM counterpart.
Main difference I’d imagine to be CPU-bound workloads, where’d we’d likely fall short.
However, a lot of message processing workloads tends more towards the IO side of things anyway, for which a BEAM implementation would fit nicely.
onkara
Yeah that’s what my intuition say too.
Yes whole problem domain seems IO bound and BEAM would be a good fit. But what I don’t understand is why is the persistence story, in all these attempts ErlBus , EventBus and Phoenix’s own PubSub, is so weak. In fact because of this incompleteness many CTO/Architects won’t consider Elixir/Erlang based solutions and default to Kafka system.
madshargreave
I’d look into using Mnesia backed by a disk-based storage engine such as levelDB, rocksDB or possibly even DynamoDB for the event log.
tristan
Phoenix’s PubSub does not do what kafka does.
I had need for a lightweight topic based log system and we created GitHub - erleans/vonnegut · GitHub
It is compatible with kafka on disk and on the wire but uses chain replication instead of Kafka’s ISR based replication. It is not ready as a kafka replacement, There is plenty to still do around membership and concensus but the internals are there. Figured I’d mention it in case anyone wanted to work on such a thing and it could be a useful base layer
onkara
@tristan thanks for sharing your contribution I will give it a try. So apart from membership and consensus is there a list of TODO items or roadmap that will nudge this to production ready release?
onkara
That begs the question, why not just use CouchDB for event logs? seems like its good alternative to Kafka