On looking at all of these benchmarks…they are all sequential.
How has this gone overlooked for so long? Not a single additional process is being used across these benchmarks?
brightball
OvermindDL1
Like really? Why not send_json since it will convert the map to JSON automatically using the phoenix registered JSON encoder (which might not be poison, and thus could be faster than poison)?
I see a lot of various things just wrong with this code… o.O
1
sasajuric
Author of Elixir In Action
If multiple processes are used, you could also encode each piece with poison in each process, and then append encoded strings to an iolist in the request handler process.
Also, using encode_to_iolist! instead of encode! should be faster.
Bypassing changesets in update might also help. Of course, it would be best to avoid read followed by the write, which is possible for the update task, but forbidden by the task rules 
1
anthonyb
Therefore, a generic framework comparison is IMO not going to tell you anything useful about your possible bottlenecks.
I don’t mean to be arguing heavily in favor of the usefulness of the TE benchmarks in particular, so I don’t want to belabor the point, but again, these statements come off as rather extreme.
Suppose the only information you had was at TechEmpower Framework Benchmarks. Are you saying you would conclude that Symfony2 is just as likely to outperform Phoenix as the other way around in a real-world scenario involving requests that query a database and return HTML?
It’s not practical to implement a real-world version of a given application in 100+ different technology stacks, so various published performance results might at least help narrow down the likely candidates for further investigation.
The middle ground IMO is checking whether a tech stack is fast enough for the kind of load you expect in the kind of system you want to build.
First, how do you check without building the system? Second, being “fast enough” isn’t always the relevant question. Suppose you implement a proof of concept in some stack and estimate that based on its performance and hardware needs, your monthly costs would be $50,000. Is that “fast enough”? What if there is another stack that is so fast it could get the monthly costs down to $5,000. In that case, you might conclude the first system was not “fast enough,” but you wouldn’t know that without having some points of comparison. But you can only build so many proofs of concept – before starting, it might help to have some existing data to point you in the right direction.
Unlike TE benches, the Phoenix test doesn’t compare Phoenix to anything else. The test merely proves that the stack can handle 2M simultaneous connections. In other words, it demonstrates the capacity of the stack, which is definitely useful to know.
I’m not sure I follow. If you want, you could look at any given framework in a set of benchmarks to determine the capacity of that particular framework, without making any comparisons to other frameworks included in the benchmarks. I don’t see how running multiple different frameworks through the same set of tests makes the tests for any individual framework any less useful.
Also, the 2 million websocket connection test was on a $1500/month machine. How do we know if that is good performance? Could we get the same performance on an $80/month Digital Ocean VM using Express? No, but we couldn’t know that without making comparisons. Without points of comparison, the single test in isolation isn’t particularly meaningful or useful.
1
idi527
It has been repeated on a $640/month Digital Ocean VM [0]. But it still doesn’t compare well to uWebSockets which would probably handle 2m websocket connections on a much smaller machine.
1
brightball
I remember seeing a breakdown of the channels benchmark that made the point that Channels uses 3 elixir processes for each socket to handle failure, reconnects and state properly. It’s not a raw “how many websockets can you hold” example.
Since the ram limit is essentially the socket limit, a raw socket test should be closer to 3x higher.
UWebsocket is another example of subbing out to C. Good looking library though.
1
idi527
I’m not sure but I think n2o [0] uses one process per websocket connection to handle all those things. So there are always ways to optimize. It’s similar to the difference between cowboy and elli, which shows that for simple use-cases sometimes it’s better to keep everything in the same process (cowboy creates a process for each connection and then also for each request, elli keeps it all in one).
[0] GitHub - synrc/n2o: ⭕ N2O: Distributed WebSocket Application Server ISO 20922 · GitHub
1
brightball
We’d need somebody with more expertise than me to speak into the channels implementation and why it’s built the way it is. @chrismccord want to chime in? 
1
sasajuric
Author of Elixir In Action
What I’m saying is that I can’t conclude anything reliably from that benchmark. That’s my main point I’m arguing in this thread.
When I first evaluated Erlang, I did a quick, dirty, and a very simple simulation of the target system, issuing a 10x of the expected load for the duration of 12 hours. I also did a few minor experiments, just to make sure I can do some typical stuff, such as talking to the database, working with XMLs, and such. I needed to see first hand that I can handle the desired load. Once I established that, I didn’t care much about few microseconds here/there.
Sure, but the thing is that raw speed is usually not the only, nor the most important factor. Once you have the tech which can handle your load, other things start to matter, such as the support for fault-tolerance, stable latency, and troubleshooting a running system. Erlang/Elixir excel at this, and that matters, because it improves the uptime and availability, and makes the life of developers much easier. I find this important, because the cost of downtime and the cost of developers are IME much higher than the cost of hardware.
Moreover, with proper algorithmical and technical optimizations, in most cases the observed speed between two different stacks usually becomes much smaller than in contrived benchmarks, such as TE. In particular, when it comes to Erlang, based on the fact that I worked with it for 7 years, and that others have done wonders with it, I’m confident that it will suffice in the vast majority of the cases. In the cases when it doesn’t (e.g. heavy number crunching), I can always step outside and reach for say, C, Rust, Go, or some other more performant language.
That might be true, but one big problem I have with TE is that they put the ranking list right in your face. It’s literally the first thing they show you. So there’s a big implication that a higher positioned stack is necessarily better.
Another problem is that I think the tests themselves are shallowly executed, which has also been argued by others in this thread.
Finally, the tasks themselves seem quite contrived. Take a look at the updates task. My first idea to optimize this would be to try to update everything in a single round-trip to the database. If that didn’t work, I’d at least remove the needless record read. But that’s not allowed by task rules. Which makes the task highly contrived IMO.
There’s a whole other bag of tricks we can throw at the problem. They usually come with some associated trade-off, but those trade-offs can only be evaluated in real life (which is not TE). Caching is for example not allowed, but that’s one of the main optimization techniques. Since you can do easy caching in Erlang without needing to run an additional process and serialize to/from json, it could do wonders for the perf, especially compared to many other techs. But that’s not tested at all in TE. So what can I conclude from the results of the TE bench? For me, the answer is: nothing Not even about performance, let alone about any other properties of a tech stack.
Assuming that you pay some developers, administrators, support, and others to manage the system with such large userbase, 1.5k/mo doesn’t seem like a significant cost in the total expenses sheet. You could get bigger savings by choosing a technology which allows developers to efficiently and confidently manage that kind of load, and to keep the system stable and running as much as possible, and to reduce the load on the support team. Again, raw performance is just a part of the story. It matters, sure, but up to a point. You need to balance it with other properties.
Also, in most cases, I don’t expect a dramatical difference between Erlang and other technologies in terms of hardware costs. While for some cases (e.g. computation heavy tasks) Erlang can be an order of magnitude slower, that doesn’t necessarily mean that you can save an order of magnitude in terms of hardware cost, unless the 100% of what you do is crunching numbers.
4