Just watched the ElixirConf2017 talk elixir with kafka, where genstage/flow is compared and used together with kafka.
I have a question: why not extend genstage/flow so that we don’t depend on kafka?
Any thoughts?
In my experience, whenever I write anything on BEAM talking with a Message Queue of any kind, it is to exchange messages with foreign systems.
Those foreign systems often do already have the ability to talk to an established message queue, while you would need to implement such stuff on your own when trying to build it on your own.
what happens If the foreign system is built in elixir?
I wouldn’t rely on that.
I’m working in a field where components are changing all the way, only constant is the message queue (its rabbit for me).
If though you can guarantee that the other components will be on the BEAM forever and they will always run on the same OTP and ERTS version, of course it is safe to replace the message queue by something “native”, but that has to be planned well and thought through a couple of times.
1 Like
I have a question: why not extend genstage/flow so that we don’t depend on kafka?
The fundamental problem genstage/flow was designed for was interacting with external systems.
The better your language is at interacting with existing systems, the easier it is to get in the door
at large ( or small ) enterprises. While it is always tempting to “ALL THE THINGS” with your shiny new toy, it is generally much better to concentrate on the thing that your new language/tool does really well and work on inter-operability with existing solutions. If your work already does Kafka, the chances of your new project getting off the ground with "First we replace Kafka with X… " are slim and none.
For good or ill, there is a huge amount of time and effort put into Kafka to solve a very difficult problem. Re-inventing that wheel in Elixir might be fun, but it doesn’t advance the cause very much. “Small” languages without major vendor support need to sneak in the backdoor and quietly get the job done. When the business discovers the teams using Elixir are doing twice as much work, with half the servers and engineers, then you might be able to work on building major infrastructure replacements in Elixir.
1 Like
