What would it take to create Kafka like solution in Elixir?

I’m not really familiar with k8s, but I have a hunch that some form of it could be implemented in beam, and exposed as a library. I think this would require having something similar to etcd, which would in turn require a solid raft library. ra seems like a promising option here.

This is pretty much my thinking too! It’s worth noting that there are things which are simpler to do at the OS level. For example, partial deploy (update only one part of your system) is much simpler with std. microservice (just deploy and restart one microservice). Technically, this could be done with beam too, but it’s more tedious. Also, at the OS level, you can have two different microservices use different versions of the same dependency, which won’t work in beam.

But either way, I think that we could get solid approximations of these heavyweight products which would work just fine in simpler situations, and it’s worth keeping in mind that there are many companies that operate on a relatively small scale and deal with much less complexity than Twitter, Facebook, Netflix & co.

5 Likes