What would it take to create Kafka like solution in Elixir?

I personally think it makes a lot of sense to implement lighter versions of such 3rd party products as BEAM libraries, with the goal of simplifying the operations and reducing the amount of moving parts. There are already many examples where we can opt for BEAM libraries instead of external tools & products, such as nginx, cron, or redis. The alternatives from the BEAM ecosystem don’t necessarily match these tools in terms of features or performance, but in many cases they can work just fine, and help simplifying the system architecture.

I’d like to see the ecosystem growing further in this area. For example, I’d love to see a relational database as a BEAM library. Something I can add as a lib dependency, start an instance (or multiple instances) somewhere in a supervision tree, and have SQL based persistence without needing to manage a separate database instance, roles, handle language ↔ db type mapping etc. When I occasionally mention this during my talks, I get some skeptical feedback along the lines of “Why would you want to reimplement databases such as PostgreSQL, MySQL, etc.?”. The point is not to reimplement or compete with established databases, but to have a lightweight alternative which would be more fitting in simpler scenarios.

Ideally, if I want to build a small to medium web-facing CRUD, I should be able to start using nothing but Elixir, and get the basic skeleton working within 15 minutes or so, with everything implemented in a single language (say Erlang and Elixir), inside a single project, running as a single OS process per each node in the cluster. As long as we’re not able to do this, I think there’s a lot of potential for improvements in our ecosystem, and implementing alternatives to established products makes a lot of sense :slight_smile:

29 Likes