Apache Kafka also promise about similar benefits as BEAM and OTP offer like Distribution, Fault Tolerance etc.
Edit: Apache Cassandra which is AMAIK a database, also promise things like Linear Scalability, High Availability and Fault Tolernace. (Because it’s database/data-storage, we might better compare it to ETS or Mnesia). End-Edit
How are these different/better/worse than BEAM and OTP ?
I don’t know if you can compare apache kafka to Erlang / OTP …
Apache Kafka is advanced / distributed, persisted transaction log (append only).
Do we have something like this in Erlang ?
But you will need big infrastructure to just maintain Apache Kafka alone.
It make sense to use it if you handle a lot of traffic with a lot of data.
Beam OTP is fault tolerant actor model
You could write application in Erlang/Elixir that could produce or consume data from Apache Kafka.
If you want to learn more about apache kafka it has nice blog https://www.confluent.io/blog/
When combined, the Kafka ecosystem at LinkedIn is sent over 800 billion messages per day which amounts to over 175 terabytes of data. Over 650 terabytes of messages are then consumed daily, which is why the ability of Kafka to handle multiple producers and multiple consumers for each topic is important. At the busiest times of day, we are receiving over 13 million messages per second, or 2.75 gigabytes of data per second. To handle all these messages, LinkedIn runs over 1100 Kafka brokers organized into more than 60 clusters.
There is also nice feature, apacha kafka can connect to existing database, and all changes on database will be streamed as events to apacha kafka.
Casandra is written on top JVM so can be sometime slow due to garbage collector.
There is alternative written in c++ and compatible with Casandra but 10 x faster https://www.scylladb.com/
So I am quite sceptic using JVM for database purpose.