Elixir and Erlang provide powerful mechanisms for building systems that are always available. But when it comes to building distributed systems that need consistency Erlang leaves many of the implementation details up to the programmer.
In this talk we’ll discuss the trade offs between systems that are always on and systems that are consistent. We’ll look at the types of guarantees that standard Erlang tooling gives us as well as their shortcomings. Finally we’ll look at several patterns and techniques for building consistent, distributed systems using elixir.
Have you looked into other raft implementations on beam, like rabbitmq/ra or maybe dreyk/zraft_lib? What are the main differences of :raft from those other implementations?
I actually think the work you are doing will be the enabler of killer apps for Elixir ecosystem especially in data processing pipelines. Among other things it will enable a Elixir native kafka like system that can expose a GenStage producer with no external dependencies on ZK. That would make Elixir the killer platform for data processing pipelines.
Even after all of my searching I hadn’t seen zraft so I’ll have to look into it more. One of the design characteristics that I wanted was a backend that used an established database engine instead of handling files directly. Thats why our implementation uses rocksdb (although since its pluggable you could really use whatever you want). Based on a quick glance at zraft it looks like its managing the log file directly. That doesn’t mean that its a poor implementation by any means. Just something I noticed and probably would have been a deal breaker for my uses. Hard to say though so I’ll have to dig in more.
Ra seems great but its under heavy development and is a sufficient departure from the original raft algorithm in regards to its failure detection that I felt better about building a solution that was a little closer to the original specification. Ra has made a bunch of tradeoffs that make total sense for its use in rabbitmq. Those are reasonable tradeoffs for them but aren’t tradeoffs that I need.
