Techempower benchmarks

I was looking at these benchmarks too the other day. And I also noticed the high error rate on Phoenix’s benchmarks. It seems strange to me, like their server wasn’t setup right. It seems implausible that Phoenix could perform so poorly. The GitHub repo is here - I thought about pulling it down and running the benchmarks myself. For one thing, the version of Phoenix is out of date, so we could make a pull request to update those dependencies.

Agree about the extremely high error rate suggesting they have set something up wrong. You’d think they’d investigate first before publishing

Many of the packages there are entirely by outside groups; that is part of the appeal to me actually, that those evangelizing their favorites can actually submit improvements.

I saw on twitter a response from Chris McCord somewhere (Twitter, Reddit, HN, ?) that stated there were serious problems with the configuration and that they had submitted a PR to fix some issues and nothing had been done. Also there was some caveat to the numbers posted where they had their hardware decommissioned and scrambled to get it published which meant the frameworks got no time to tweak/fix stuff.

I would say the jury is WAY out on the legitimacy of the numbers.

I believe this is the comment you’re talking about:

Any reason why Elixir/Phoenix did so badly in Round 12 of Techempower benchmarks?

We don’t know what caused the errors and unfortunately we didn’t have a chance to collaborate with them on a true run. A few months ago they added Phoenix in a preview, but it was a very poor implementation. They were testing JSON benchmarks through the :browser pipeline, complete with crsf token generation. They had a dev DB pool size of 10, where other frameworks were given of pool size of 100. And they also had heavy IO logging, where other frameworks did no logging. We sent a PR to address these issues, and I was hoping to see true results in the latest runs, but no preview was provided this time and we weren’t able to work with them on the errors. tldr; these results are not representative of the framework.

169,526 errors?! Based on those benchmarks I might as well go back to Codeigniter.

omg… I seriously lol’d. My wife was like WTF?

Hot off the presses in the round 14 battle - TechEmpower Framework Benchmarks

Phoenix somewhere at the median line. I am not too crazy about these numbers. Just interesting to watch.

Nothing about that looks right