Looking through some of the other tests, it does look like many are taking steps to ensure that the same database connection is used to burn through the entire set of requests. Doesn’t Ecto hand it back after every call in the current setup? Is there a way to avoid that? Ecto.Multi maybe? I saw the tests for Ruby using Sequel were all wrapped in DB.synchronize.
brightball
0
cmkarlsson
I don’t mind benchmarks. They can be useful. But I don’t like how most of the benchmarks out there are done because they measure the wrong thing!
I think TE benchmarks have the right idea. They test against more than just the HTTP layer (i.e database, data serialization, etc) but they measure the wrong thing. They have the wrong test setup to give you an idea.
And phoenix and erlang does great in these benchmarks! If you analyze the results some of the benchmarks that beat phoenix did it once. For a particular 15 second period, for a particular (low) concurrency level.
Unless the framework consistently give good result for the various concurrency levels and benchmarks something is wrong with either the test or the framework and I would discard the result at once. If the results are jumping all over the place I would discard the result. A max latency that is too high is also something I would consider a warning flag.
If you sort by max latency phoenix is:
- phoenix is #6 for json
- phoenix is #2 for single query
- It drops down a bit on the others but still higher than the average (why do they use average? it should never be used.)
They test a closed system and not the open web. This is a very important distinction. Testing against a closed system and comparing with an open system means you will have the “coordinated omission” problem. I.e the client and server work together to make sure the server isn’t overloaded. The open web doesn’t work like this. Clients will not stop sending requests because another request is slow.
If someone wants to re-run the TE benchmarks please:
- Client framework. Either use full simulator (I think tsung will work here) or use a benchmark tool that can work with coordinated omission (wrk2). The difference is massive!
- Use much higher concurrency. For the open web, if you are serving 300,000 request per second I can tell you that your concurrency is deemed to be somewhere along the 100,000 mark at least. They test with 256.
- Run the tests for longer time.
- Display the full data tables, no summaries.
Run these tests again with the above parameters and I will assure you phoenix will end up higher currently and it should generally outperform any ruby/python/php framework out there with a big margin.
If you ask me: If you test any other way for the open web you are wasting your time and optimising the wrong thing.
4
DianaOlympos
In general, the feeling i get for having followed multiple round of tentative by phoenix people to work on the TE benchmark :
- It is really really hard to understand what is happening in these benchmark
- it is really really hard to get anything that look like perf data that would allow to analyse and solve what makes something slow
- The TE team tend to modify the code of a particular framework when they want, breaking it regularly
- In general, getting anytime of information from the TE team is super hard.
So if you want to fix it, go for it. But i prefer to fix a complex problem or help build the next Orleans for the BEAM. It will results with far more net result for the time invested.
3
sasajuric
Author of Elixir In Action
Yeah, I believe that this selling point is poorly worded. The “speed” is a very relative thing. It depends on your needs, and your requirements. Sometimes you don’t care about milliseconds, other times a single millisecond can be an eternity.
I’d say instead that Erlang/Elixir/Phoenix doesn’t require a trade-off between productivity, fault-tolerance, and scalability, and will give you a “reasonable” speed in many cases, with the option to easily step outside of Erlang for the situations where Erlang simply doesn’t suffice.
By studying properties of the runtime and the language, and understanding what tools does it give you to help you get those properties.
For example, in Erlang, a process crash doesn’t disturb anything else in the system, but any other process can be notified about it. This gives you a way of isolating crashes but also to detect them and respond to them, thus allowing your systems to self-heal. Shared-nothing concurrency ensures that the failing thing doesn’t leave the garbage behind.
In contrast, in go, an uncaught error (panic) in a goroutine takes down the whole OS process. Shared memory means that even if you catch that error, you might still end up with an inconsistent global state, leading to other errors. Improving fault-tolerance, while certainly possible, is going to be harder in go, simply because the language doesn’t give you the kind of support Erlang gives you. This shouldn’t come as a surprise, given that Erlang was built precisely for systems which run continuously and experience as little downtime as possible.
For a few other interesting and unique properties of BEAM, you can also take a look at my ElixirDaze talk.
The individual measurements can give you some hints about how you’re doing. They can also help you check whether you’re improving or degrading in performance, as you make subsequent changes. This is where a number is good. If today it’s much higher than it was yesterday, then likely some change had affected performance in a bad way.
In contrast, bench comparison is a very dangerous territory, because you end up reducing a stack to a number, and then choose one based on comparing numbers, completely disregarding the features it gives you.
I wasn’t suggesting reducing this property into a number. Instead, I’m saying that we should focus more on how the tech helps us with some difficult challenges. Fault-tolerance is one such challenge, which is both required and difficult even in a moderately complex system. Therefore, the question is how can tech help us deal with this challenge from the day one, when things are always simple, all the way to production when things are never simple.
That’s definitely not an exact science, so it’s always a bet, but a deeper look into what the tech gives you can definitely help you make a more educated guess. In contrast, a simple comparison of two numbers is IMO as good as throwing a coin 
I’m not suggesting doing it for every project. I did it once, the first time I considered Erlang. Once that was followed by the success in production, and my deeper understanding of the tech, I got a much higher confidence about the stack.
2
sasajuric
Author of Elixir In Action
I forgot to address this point, so just want to confirm that this is precisely what I’m saying. Regardless of the fact that I believe that choosing a stack amounts to way more than focusing on raw perf, my impression is that TE benches are fundamentally flawed as well as poorly conducted, so I think that their results have little correlation to what would happen in a real world scenario.
2
anthonyb
By studying properties of the runtime and the language, and understanding what tools does it give you to help you get those properties.
Sure, but then the question becomes how much do these properties matter in practice (i.e., how much more reliability do they really buy, and at what cost)? Ultimately, I think you need to see real measurements to confirm expectations based on analysis. An analysis of the properties of the runtime would be less compelling if we couldn’t also point to examples like the famed nine nines of uptime. You indicated that in your own initial evaluation of Erlang, you tested it out – you didn’t pick it blindly after merely analyzing its properties.
In contrast, bench comparison is a very dangerous territory, because you end up reducing a stack to a number, and then choose one based on comparing numbers, completely disregarding the features it gives you.
Sure. That doesn’t mean they can’t provide value when used appropriately. I suppose the same argument could be made about comparing systems merely based on their properties/features – a particular feature may sound compelling on paper, but if it can’t be demonstrated to tip the cost-benefit scale in the right direction in practice, it has no real value.
0
kseg
Sounds to me like you’re a fan of the benchmarks that show Phoenix in a good light and not so much those that show it in a bad light.
The 2M connection is basically testing Linux. It opens a socket, accepts connections and reads a ~1KB. The most basic TE benchmark is more representative of HTTP load than that is of websocket load. Apparently when a benchmark “demonstrates” that Phoenix doesn’t add much overhead, that’s a good benchmark. If it demonstrates that it adds a lot of overhead, well..the benchmark is flawed, or you should care more about stability, or it doesn’t represent real world examples.
0
kseg
Wrapping the outer handler in a try/catch is neither difficult nor a challenge. Which is why net/http does it: go/src/net/http/server.go at master · golang/go · GitHub
There’s two groups of people in this conversation about performance: those that think performance could be improved and those that want to talk about everything except performance.
0
chrismccord
Creator of Phoenix
This is just not true. Our Chanel benchmarks do more than just accept a connection. We exercise the channel protocol which spawns and isolated, concurrent process monitored by the underlying websocket transport, which is subscribed to our pubsub layer. We also put the pubsub layer thru its paces by broadcasting messages to the 2M clients.
This is not a fair point. Saša has done nothing here to do mental dynamistics to make up for any apparent elixir or phoenix deficiencies. For anyone remotely adept at writing Elixir, the techmeme app is obviously poorly written, from configuration to middleware to lack of use of concurrency in the db tests, and more. My experience with them over the last two years has given me little confidence to see value in their results.
I’ll also note that this thread is turning unnecessarily negative, so please take care to avoid things going toxic
9
chrismccord
Creator of Phoenix
It’s designed this way because channels provide:
- isolated, concurrent lines of communication on the underlying multiplexed transport process. You don’t want an error in one channel to bring down the others. Likewise, you don’t want one channel to block others if doing any blocking or intensive work. Put simply, it’s designed exactly how you’d manage other system sin the runtime – isolated concurrent processes that are supervised.
- client reception of channel errors. If we used a single transport process and one of your channels caused a crash, you’d take out the physical connection. With our setup, clients get notified of channel crashes without taking out the connection and can rejoin the channel without the extra trip to reconnect.
Yes these things require 3 processes vs 1, but our 2M benchmarks exactly show that this overhead scales. Phoenix also allows you to write your own transport that would forgo this multiplexed model. @sasajuric’s has also done recent transport work to make things more extensible for these scenarios. That said, for the vast majority of usecaes our default approach is the way to go.
7