How would you compare Kafka to BEAM and OTP?

Apache Kafka

I don’t know if you can compare apache kafka to Erlang / OTP …

Apache Kafka is advanced / distributed, persisted transaction log (append only).
Do we have something like this in Erlang ?

But you will need big infrastructure to just maintain Apache Kafka alone.
It make sense to use it if you handle a lot of traffic with a lot of data.

Beam OTP is fault tolerant actor model

You could write application in Erlang/Elixir that could produce or consume data from Apache Kafka.
If you want to learn more about apache kafka it has nice blog Confluent Blog | Tutorials, Tips, and News Updates

Use Case

When combined, the Kafka ecosystem at LinkedIn is sent over 800 billion messages per day which amounts to over 175 terabytes of data. Over 650 terabytes of messages are then consumed daily, which is why the ability of Kafka to handle multiple producers and multiple consumers for each topic is important. At the busiest times of day, we are receiving over 13 million messages per second, or 2.75 gigabytes of data per second. To handle all these messages, LinkedIn runs over 1100 Kafka brokers organized into more than 60 clusters.

There is also nice feature, apacha kafka can connect to existing database, and all changes on database will be streamed as events to apacha kafka.

Apache Casandra

Apache Casandra is distributed database. I assume you can compare some distributed database written on top of Erland like Riak Distributed Databases | Distributed NoSQL Databases | Riak vs Casandra
Riak |

Casandra is written on top JVM so can be sometime slow due to garbage collector.
There is alternative written in c++ and compatible with Casandra but 10 x faster https://www.scylladb.com/
So I am quite sceptic using JVM for database purpose.

Recommend our topic :slight_smile:

4 Likes