How would you compare Kafka to BEAM and OTP?

General Prog (Archived)
1

Apache Kafka also promise about similar benefits as BEAM and OTP offer like Distribution, Fault Tolerance etc.

Edit:
Apache Cassandra which is AMAIK a database, also promise things like Linear Scalability, High Availability and Fault Tolernace. (Because it’s database/data-storage, we might better compare it to ETS or Mnesia).
End-Edit

How are these different/better/worse than BEAM and OTP ?

2

Apache Kafka

I don’t know if you can compare apache kafka to Erlang / OTP …

Apache Kafka is advanced / distributed, persisted transaction log (append only).
Do we have something like this in Erlang ?

But you will need big infrastructure to just maintain Apache Kafka alone.
It make sense to use it if you handle a lot of traffic with a lot of data.

Beam OTP is fault tolerant actor model

You could write application in Erlang/Elixir that could produce or consume data from Apache Kafka.
If you want to learn more about apache kafka it has nice blog Confluent Blog | Tutorials, Tips, and News Updates

Use Case

When combined, the Kafka ecosystem at LinkedIn is sent over 800 billion messages per day which amounts to over 175 terabytes of data. Over 650 terabytes of messages are then consumed daily, which is why the ability of Kafka to handle multiple producers and multiple consumers for each topic is important. At the busiest times of day, we are receiving over 13 million messages per second, or 2.75 gigabytes of data per second. To handle all these messages, LinkedIn runs over 1100 Kafka brokers organized into more than 60 clusters.

There is also nice feature, apacha kafka can connect to existing database, and all changes on database will be streamed as events to apacha kafka.

Apache Casandra

Apache Casandra is distributed database. I assume you can compare some distributed database written on top of Erland like Riak Distributed Databases | Distributed NoSQL Databases | Riak vs Casandra
Riak |

Casandra is written on top JVM so can be sometime slow due to garbage collector.
There is alternative written in c++ and compatible with Casandra but 10 x faster https://www.scylladb.com/
So I am quite sceptic using JVM for database purpose.

Recommend our topic :slight_smile:

4 Likes
3
HomeAway uses Confluent & Apache Kafka® to Transform Travel

double digits billion messages per day

https://www.homeaway.com

The future of reactive applications is real time stream processing …

4
Confluent offers Apache Kafka as a Service in AWS
5

ETL Is Dead, Long Live Streams: real-time streams w/ Apache Kafka