And this is where I think is the core difference between our opinions.
I don’t think it is so much that we disagree, but that to some degree you have been caricaturizing my arguments and somewhat inconsistent in your own.
Please note, my initial post was in response to the following comment from you:
If we could have an easy way of installing TE suite locally and running, we could have an automated way of benching our frameworks, finding out possible bottlenecks, and even detecting regression performance. Not sure to what extent can TE code currently be used for that, but I agree that this can be useful.
So, initially you seemed to acknowledge value in making use of the TE suite for measurements. I was just pointing out that it is indeed fairly easy to get the suite up and running on a local machine with the provided Vagrant setup. You could probably be fully set up and ready to play with the code in about 30 minutes. Furthermore, the endpoints are quite simple, so I would think an experienced and knowledgeable Elixir/Phoenix dev could quickly make improvements if the current code is not as idiomatic or performant as it could be. If nobody cares about what Phoenix looks like in the TE data, that’s fine, but if there’s any interest in making improvements, we’re probably talking about a time investment of a few hours at most.
You seem to think that you can somehow assign a number to foo and bar, and then pick one based on that number,
I don’t think I have said anything like that. When choosing a technology, many attributes may be relevant to the decision. Performance (speed) may be one of those attributes. Reliability may be another attribute. Both of those things (among many others) can be quantified in various ways, and our decisions should be better to the extent that we have high quality measures of attributes that matter.
The question is however, how much a particular tech helps you with that. My experience is that Erlang helps a lot, thus making that difficult job easier. However, I cannot quantify that into a number.
But you can quantify that into a number, and implicitly, you have. When you say “Erlang helps a lot,” there is an implicit comparison there – you are saying it saves you time, effort, headache, etc. compared with other options you have tried or heard about. If it doesn’t really save you time and cost over other options, then why prefer it and recommend it? You may not have precise measurements of the benefits and cost savings, but (a) you at least have a sense that they are non-trivial (i.e., they exceed some threshhold), and (b) they could in principle be measured more precisely, and it would be useful if they were.
Yes, but how do you know if Elixir exhibits better fault-tolerance, stable latency, and uptime compared with other options?
By studying properties of the runtime and the language, and understanding what tools does it give you to help you get those properties.
You indicated that in your own initial evaluation of Erlang, you tested it out – you didn’t pick it blindly after merely analyzing its properties.
Precisely, but the point of that test was to verify that Erlang is capable of handling my load. It wasn’t a comparison of Erlang to other stacks, but merely a verification that the stack is capable for what I needed.
The above exchange is another example where you appear to be disagreeing with me but have in fact simply restated my point. I started by asking how you know Elixir is more reliable, and your initial response was by mere analysis of its properties, independent of its actual real world behavior. I pointed out that you also need to verify any analysis by testing actual behavior, and that you in fact have done so yourself – and now you seem to be agreeing with this point.
Personally I think these things are hard to properly quantify. However, looking at the properties, you can at least decide which ones do you value. For example, Erlang has excellent support for fault-tolerance, but is unspectacular in CPU processing, and terrible at crunching numbers. Knowing that, I can pick or avoid Erlang, depending on the particular specifics of the problem I’m solving.
How can you pick without making at least some rough attempt to quantify? Let’s say you need to do a lot of CPU processing? How do you know whether Erlang is or isn’t as fast as any other language you might consider, or whether the differences really matter in terms of cost, etc.? Presumably at some point you are attempting to make some measurements or relying on measurements others have made.
Finally, to be clear, I am not defending the TE benchmarks, and I think you have made many good points about the qualities of Elixir/Erlang and the many factors to consider when evaluating technology solutions. Thank you for sharing your experience and insights.
