Not really, Kafka is persistent, Elixir’s facilities like ETS are ephemeral.

Elixir and Erlang don’t eliminate the need for Kafka, what happens is that in some use cases for Kafka in other stacks you could handle without Kafka.
Use cases that are related with pub/sub and data ingestion could be easily done without Kafka just by using the phoenix pubsub or broadway.

Stuff that relies heavily on Kafka persistent storage isn’t the same thing. Also there are a lot of conectors to Kafka that can make it some solutions possible that you would need to implement manually on a Elixir/Erlang system.

I’d say that you can avoid the complexities of running Kafka and use tools available in Elixir/Erlang to architect your solutions differently. It’s a matter of trade offs, and Kafka is a big compromise given the required infrastructure and the need for specialized knowledge around it. It’s not Kafka, but a good example of approaching problems differently to simplify your architecture you can check on this talk

Thanks for the responses!

Hmm, I did read a bit about ETS but it was more in the context of providing an alternative for memcached or redis for the most common use case, which is caching.

Amazing! I had heard of Phoenix before, and was quite impressed by some of the stuff that can be done with LiveView, but I had never heard of broadway. I’ll definitely look more into it.

I’m wondering if you have any examples of such things?

Oh yeah, this is one thing I’ve had in mind a lot lately. I’m currently working in a team where, I hate to say it, but the infrastructure peeps are not the most knowledgeable (and neither am I!), so I’d love to work in such a way that I have to rely on the least amount of components possible and still be confident it can scale without having to spend a fortune (which is the solution they’ve used at previous jobs where everything ran their entire backend on Python and Django… they just threw money at the problem!). Basically I want to make our lives as easy as possible… TYSM for the video, I will definitely watch it as well.

True, my point is that if you want to go beyond GenServer mailboxes – which is the thing that can be a mini-Kafka for many scenarios – then people opt for either ETS for in-memory storage, or DETS / Mnesia / actual DBs for non-volatile storage.

I do agree with @cevado here (and I don’t like Kafka myself) but truth is that our business requirements don’t allow us to forget events or messages so we need something battle-tested.

All of what he showed you, and what you can find yourself after, is true and it exists and is super useful, but you still will need persistence at one point. That’s what I was getting at.

Hi!

I’m also an Elixir newb.
Would Oban be a good fit in regard to persistency?

What use cases can’t Elixir + ETS + GenServers + Oban cover, that Kafka could?

Feature-wise they give you almost anything you might need. Kafka and the like (Redpanda, NATS etc.) give you ultra bandwidth; we’re talking 150k+ messages per second.

Not many apps need that though.

just making things clear:
Oban is a background job tool, it uses db to manage jobs, but to keep it performant and in a healthy state the data on jobs needs to be ephemeral, at some point in time you gonna delete the data of jobs already executed(with success or failure).
Kafka on the other hand is an event store, built do deal with massive amount of events.

It’s good to remind that ETS is an in memory key value store and GenServer is an abstraction of a processing worker.

Just given the nature of those tools any multi-services event source/event store solutions would never be possible with only oban, ets and genservers. But I personally would avoid this type of thinking, at least to what comes to engineering solutions, it’s better to start with a problem and think of multiple ways to get to what needs to be done.

Sorry I missed this reply. But here is a list of kafka connectors(stuff that can work as data source or data destination of events in a topic):