I have had multiple conversations with Dave on multiple topics in the past. I would say the best summary of those discussions is that I agree with Dave on the problems but we often disagree on the solutions.

For example, I agree on the problems the Jeeves project aims to solve. It is really nice to have some state and then being able to pool it, partition it, or whatever based on a single line of code change. Those things are very annoying to do today and Jeeves aims to solve it. However, Jeeves is a proof of concept and it has other areas that require improvements, such as its over reliance on macros. I don’t mean this as a criticism at all, my point is exactly that figuring out a complete model is a lot of work and we - as a community - should probably have done a better job of encouraging this type of experimentation. I also think there should have been more discussions on the problems rather than solutions, because when we jump into solutions, a lot of people’s knee jerk reaction is to say “there is nothing wrong with GenServer”, which I agree is very discouraging to hear when you are trying to improve it, but it also means a lack of agreement on the problem statement.

There is a cognitive dissonance, where there is a language that is extensible, but there is a conservative community that would rather not extend it. I believe most of the problems Dave mentioned are solvable technically - you can replace how pipe works, you can compose Plugs at runtime, you can implement your own data pipeline solution, etc - but most people would prefer to not change things.

If this conservatism is a good or bad thing, it is up to the community to decide. Some communities are conservative by definition (see Go) - but this means people like Dave would rather go elsewhere - while other people will prefer the stability.

I listened to parts of it and found the elixir critique towards the end rather weird. When he was talking about pipelines he seems to have missed the concurrency/processes aspect of things. I know he hasn’t but his idea of using pipes just doesn’t fit in. Yes, you could have composable pipes with some form of currying but that doesn’t really attack the same problem as streaming data concurrently through a system. Worst case you end up with a very complex tool which could be difficult and confusing to use. And if he wants to throw the oldest stuff in the pipe then just fix it.

If you want your system to live for more than a few months then you need a more conservative base on which to build it which isn’t changing all the time otherwise it very quickly becomes unmaintainable in the long run. Tough, but that’s the way it is. And that’s not restricted to telecom systems.

And if he wants a really extensible language then he shouldn’t go with something boring as javascript. No, in that case lisp is the only way to go.

And yes I am erlanger and lisper at heart.

Agreed. Even without considering the concurrency aspect, a function pipeline is unfortunately too simplistic to model all of the concerns in a data streaming system. The Haskell community is not using function pipelines to model those either: for example, the conduit library has an explicit focus on sources and sinks, which is something similar to what you would find in Flow. In data streaming, you may have multiple sources, data repartitions, multiple sinks, and all of those lead to topologies that are by definition more complex than a function pipeline. I guess someone could apply the Flow programming model to Plug but then it would make Plug much more complex than what it needs to be. It would be similar to removing the |> operator and adding monads, because you can express pipes with monads anyway.

Once you add concurrency and error handling, it becomes even more distinct. I have been working on this problem for 5+ years, first with GenStage (which does support push and pull systems as well as using buffers for load regulation instead of back-pressure), then with Flow, which is more data-centric, and now Broadway. The difference between Flow and Broadway is exactly that once you want to take error handling into account, which is a must for ingestion systems, the error modelling needs to be to put front and center rather than an afterthought. You need to reason about what happens when half of the data fails to process, what if acknowledgements are missed, etc.