Agreed. In a benchmark like this the goal isn’t a real world test. It’s a “what can you squeeze out of it”. We had another good thread talking about benchmarks that were closer to real world though.
EDIT: Reading through the rules of these benchmarks, it looks like there might be something that Erlang/Elixir could take advantage of for a performance gain, especially on the multiple queries test. The queries are executed individually with a random id and it strikes me as feasible that if queries could be executed in a process named by query and id that we could naturally leverage the BEAM to de-duplicate in progress requests. I wonder what it would take to do that and/or if it would help.
I do agree that the word “fast” in Phoenix advertisement is quite vague and should be perhaps replaced with “scalable”. The pitch I gave at my Phoenix talks is that with the stack of Erlang/Elixir/Phoenix, you can get on board quickly, move forward at a reasonable pace, write reliable, fault-tolerant systems, and be confident that the technology can help you go very far in the case your system becomes very popular and you need to scale it. That’s a bunch of wins on all accounts, and this is why I use and recommend Phoenix.
Raw speed is not Erlang’s forte (in fact sometimes it even sacrifices the speed for some other properties), so usually a sequential code will not be as fast as in many other popular languages. But the question for me isn’t who’s fastest, but rather is the tech fast enough? IME in the vast majority of the cases, Erlang is more than sufficient for my needs.
Doing various algorithmical and technical optimizations can usually take me very far. For example, looking at the rules of the “updates” task, I’m confused with some requirements and limitations. In real life, I’d try to optimize this task by performing a single update of all the target rows, say perhaps using UPDATE ... RETURNING of postgresql. I expect this would do wonders for the perf, since in a single trip to the DB I could change everything, and get all the data (and only that data) which I need.
Another option would be to try updating from multiple processes (currently it’s done sequentially).
This is not what’s advertised on the TE site. My understanding is that they think we should pick our framework based on their graphs, which is something I strongly disagree with given that the tests are synthetic, contrived, and poorly executed. You could just as well roll a dice, fire up a RNG, or consult a tarot expert to pick a framework
These are all good points, but they end up sounding a bit like rationalizations one would expect from a framework that is not near the top of the list. Python isn’t known for its speed either, nor Ruby, nor PHP, but they all have frameworks (even full stack frameworks) beating Phoenix on some of the tests.
But the question for me isn’t who’s fastest, but rather is the tech fast enough?
Sure, but in the “Data updates” test, Phoenix seems downright slow in Round 14, though it was among the fastest in Round 13. Hopefully that can be fixed.
In real life, I’d try to optimize this task by performing a single update of all the target rows
Presumably they are trying to represent real world scenarios in which you do in fact have to do multiple separate updates, perhaps to several different tables/collections. For simplicity, rather than actually creating many different tables in the database, they just have the apps make multiple updates on the same table.
Keep in mind that I’m not a member of the core team, but just a user of Phoenix. As someone who has been using Erlang for 7 years, and Elixir for 4, in a fairly loaded system (~ 2k non-cacheable reqs/sec), I have high confidence that I can reach the desired performance numbers for the vast majority of the cases.
And this leads me back to the point I already raised in this thread. No two real-life (aka production) systems are the same, and the perf bottlenecks can appear anywhere for various reasons. In most cases I’ve seen, the bottlenecks had little to do with the actual framework being chosen. Yet, I’m supposed to pick a framework based on these synthetic and highly contrived example? That doesn’t sounds right to me.
That’s of course worth looking into. Perhaps there’s some low-hanging fruit to be picked there.
However, this significant change in the ranks makes me wonder even more to what extent are these tests accurate and useful.
I don’t think anybody here expects Erlang to be faster than pure C. You can offload parts of the benchmark to C, but doing so would eliminate all of the consistency guarantees that come from the BEAM. If anything, the fact that you get all of those guarantees and comparable top end speed with Elixir without having to substitute chunks of your code with another language is a pretty major factor.
I am genuinely curious what happened on the updates though. The benchmarks that have a real concurrency test should, in my opinion, be the biggest target.
In looking at that commit, why aren’t the updates being done concurrently? Am I reading that code wrong or are they executing 1 - q queries sequentially?
|> json(Enum.map(1..q, fn _ ->
id = :rand.uniform(10000)
num = :rand.uniform(10000)
w = Repo.get(World, id)
changeset = World.changeset(w, %{randomnumber: num})
Repo.update(changeset)
%{id: id, randomnumber: num}
end))
If q is 10, they are doing 10 get / updates sequentially?
I’m generally in agreement, but consider the following statement:
No two real-life (aka production) systems are the same, and the perf bottlenecks can appear anywhere for various reasons.
This seems extreme. Surely there are many systems that have quite similar characteristics and performance demands and for which useful insights could be gained from standardized benchmarks of alternative platforms. I’m just suggesting there may be a middle ground between “benchmarks are completely useless” and “I should pick my framework based solely on a single benchmarking test I saw.” Declaring the former in the face of a recent subpar result seems a bit suspect, particularly when the top pinned tweet at Phoenix Framework (@elixirphoenix) / X is about a different “synthetic” test where Phoenix seems quite impressive.
In any case, my larger point was just that from a marketing perspective, it might be worth looking at the Phoenix code used for these benchmarks to ensure it is not unnecessarily slow. Hopefully that makes sense, even if you don’t think the benchmarks themselves are particularly useful.
No, I wasn’t talking about the Fortunes benchmark. As noted, Phoenix does fairly well against full stack frameworks on most tests, including Fortunes. It falls down on “Multiple queries” and “Data updates” specifically, and in those cases, the Python, Ruby, and PHP frameworks beating it are not offloading the work to C.
There are many differences between any two systems, such as deployment environment, network speed, database, load patterns, the kind of job that needs to be done, guarantees you need to provide etc. Therefore, a generic framework comparison is IMO not going to tell you anything useful about your possible bottlenecks.
The middle ground IMO is checking whether a tech stack is fast enough for the kind of load you expect in the kind of system you want to build. Later on, benching your system continuously is also useful.
Unlike TE benches, the Phoenix test doesn’t compare Phoenix to anything else. The test merely proves that the stack can handle 2M simultaneous connections. In other words, it demonstrates the capacity of the stack, which is definitely useful to know.
In pragmatic terms you’re certainly right. It would be nice if Phoenix was higher on the list.
OTOH I’m personally terribly sad when I think of the amount of effort invested in TE benches, given that I find them to be worse than useless. IMO, this is a huge waste of effort. Personally, I’d rather see Phoenix team be focused on Phoenix features, stability, security, performance improvements (but focused ones, along the lines of the 2M test), and such.
