Is Ecto slower than Rails ActiveRecords?

[jeremyjh]

@cmkarlsson

It is internal to your application.

@jeremyjh

Competent and well-funded teams compile their own data about their own application in their own environments which is much more useful

@cmkarlsson

According to the TE benchmarks golang and java should completely outshine erlang but for our application it is much closer and the throughput makes such little difference that other factors (stability and fault tolerance) are more important.

That doesn’t surprise me at all, it is very consistent with other benchmarks I’ve seen (such as Riak vs. Cassandra). One thing to watch out for though is the JVM GC tuning. Erlang and Go are both optimized for latency out of the box, while the JVM is optimized for throughput. How many different garbage collectors did you try on the JVM?

@cmkarlsson

Trying to create a better benchmark is obviously possible

My point is that people are going to look at the best available data, even if it is terrible. They are going to draw conclusions, even if the purveyor of that data says you can’t make those conclusions based on the data. Complaining about this and offering anecdotes is - in my opinion - useless. What would be useful is better data.

[michalmuskala]

I guess this is relevant: https://twitter.com/ivarvong/status/948677921358647296

• 70+ requests per second all day

• two VMs
• 95th percentile response time: 17ms
• elixir 1.5 / phoenix 1.3

And https://twitter.com/ivarvong/status/948689049224609792

It’s the post rendering route, eg Justin Timberlake is rebranding as a white man | The Outline …, so it fetches a few rows from Postgres and renders templates. We don’t cache HTML responses at the CDN, which allows us to do server-side feature flags, A/B testing, etc.

The lowered need for caching is something that I’ve also experienced. It makes everything so much easier.

[cmkarlsson]

@jeremyjh

How many different garbage collectors did you try on the JVM?

Probably one but I’ll have to check with the java team to know for sure. I am not a java expert so had to read up java GC tuning a bit and I am sure there are things that can improve our java results.

@jeremyjh

My point is that people are going to look at the best available data, even if it is terrible.

Yes And it is also hard not to look for data you want to see (confirmation bias). If one wants confirmation ActiveRecord is faster it is easy to find the data to support it and of course the other way around is equally true.

@jeremyjh

Complaining about this and offering anecdotes is - in my opinion - useless.

Yes, it seems like it I just can’t stop keep trying it once it a while when these discussions come up. I guess I am just hoping that it makes someone think twice about relying too much on TE or other benchmarks.

@jeremyjh

What would be useful is better data.

Would publishing methods and benchmark results of our testing move it from anecdotal to better data? If so perhaps I need to seek approval to do that.

[sasajuric]

@jeremyjh

I think what TE is trying to do is along the right lines and shouldn’t be dismissed so easily.

I respectfully disagree. I think that TE is asking a completely wrong question.

In my opinion the question is never which tool is the fastest. Instead, one of the question should be, is it fast enough. Some other important questions might be, can it scale, is it fault-tolerant, how is the ecosystem, what is the operational support, etc.

None of these seem to be in the focus of TE. They run some synthetic benches, give us flashy graphs, and this is supposed to give us a clue about which tool can should we choose. In my personal experience, in many cases the perf differences in such synthetic benches either don’t matter in the grand scheme of things, or can be significantly reduced by with proper algorithmic and technical interventions.

In some cases, squeezing every nanosecond might matter. I personally think that such cases are special, and a team facing such challenge will get a much more informative answer by conducting their own tests against a simulation/approximation of their own use case, instead of looking at TE, or any other generic/synthetic bench.

Therefore, I personally think that TE is fundamentally flawed in its premise. I also don’t have a high opinion of its implementation, but that’s irrelevant

@jeremyjh

but if you are evaluating comparable frameworks that you don’t use today, what can you look at?

Maturity, approachability, community, ease of use, flexibility, runtime guarantees, ecosystem, scalability, fault-tolerance support, are some things that come to mind. And, yes, performance is also relevant, and therefore worth looking into, perhaps by making your own tests and seeing if the tech can deliver.

[jeremyjh]

@cmkarlsson

Would publishing methods and benchmark results of our testing move it from anecdotal to better data? If so perhaps I need to seek approval to do that.

Its a lot of work to do it well, which is why its always either very narrowly scoped, or poorly done. But yes you would need to publish everything required to reproduce your results. The code, configurations, the specification and machine types, test data, methodology etc (preferrably EC2 or other major cloud provider).

[jeremyjh]

@sasajuric

Maturity, approachability, community, ease of use, flexibility, runtime guarantees, ecosystem, scalability, fault-tolerance support, are some things that come to mind. And, yes, performance is also relevant, and therefore worth looking into, perhaps by making your own tests and seeing if the tech can deliver.

I’m sure you didn’t mean it this way, but this honestly comes off as pretty condescending. The topic of this thread is performance, but I wouldn’t be using Elixir in production if I didn’t know its value proposition beyond raw performance - and I actually I think the performance is generally very good and easy to predict.

TE doesn’t have to solve world hunger in order to address just one question from your list. Scoping their evaluation strictly to performance is totally valid, as long as they don’t say you should uncritically accept their results as the only criteria used to make a decision. And yes as I mentioned, its always better to bench your own application in your own environment. But that doesn’t mean all synthetic performance data is automatically useless. In fact I think TE has demonstrated or at least raised awareness of some bugs and regressions, that were later fixed in their respective projects.

[MikeAnthony]

@ardhitama

According to this, even go is slower than mri. Maybe something is off there.

In fairness to techempower (no matter what you think of it generally) you are linking to a page that has a big old warning at the top of the page that its preliminary and potentially defective.

[ardhitama]

I think TE provided me a very good insight on how to tune perf in Erlang and many other platforms just by using runtime switches or better config leveraging combined community effort. Hence nothing wrong with TE as company.

What I irked about is only that particular result I linked. Will be better if you can put analysis why it happened like that in accordance to banner you saw there

[MikeAnthony]

This is the link to the last finished round

TechEmpower Web Framework Performance Comparison

you linked earlier to round 15 which is preliminary and not the final benchmarks

[sasajuric]

@jeremyjh

I’m sure you didn’t mean it this way, but this honestly comes off as pretty condescending.

I’m sorry you feel this way, as I didn’t mean to sound like that nor question your view on Elixir.

My statement was a general (not Elixir specific) answer to your following question:

@jeremyjh

but if you are evaluating comparable frameworks that you don’t use today, what can you look at?

I perceive this question as a rhetorical statement which suggests that TE (or any similar generic comparison) is the only/best place to look at when comparing frameworks. Perhaps my interpretation is wrong?

Either way, if we just limit this discussion to performance, I still think that TE is essentially useless. Extrapolating and generalizing from a synthetic bench comparison is very easy, but I think it’s also misleading.

I agree that some information is better than no information. However, my opinion is that TE data is misinformation, because of the way the bench is conducted, and because of the way it is postulated. And in my book, misinformation is worse than no information.

@jeremyjh

But that doesn’t mean all synthetic performance data is automatically useless.

I’m not against synthetic benches conducted in isolation. In fact, I occasionally reach for those in practice, and in my posts, such as here and here.

My problem is with comparison benches, and extrapolated conclusions that arise from those benches. If you take a look at the mentioned posts, I try hard to avoid drawing such conclusions. This is particularly obvious in the second post, where I do an Elixir take on Pusher’s bench of maximum observed latency in Go. Even though I got much better results in Elixir, not once have I compared it to Go. That post is not about Elixir vs Go, it’s about what can we do with Elixir to reduce the tail latency.

This is how I personally think about evaluating performance in general. It’s not about trying to win a battle of numbers, it’s about trying to achieve acceptable property in the system. YMMV of course

[ardhitama]

From elixir code, it should be pretty clear why that thing is slow.

They are benchmarking Poison as JSON encoder as well under DB Query benchmark, which I believe we already aware that Poison is not very performant.

One quick fix is to change the JSON encoder.

[jeremyjh]

@sasajuric

I perceive this question as a rhetorical statement which suggests that TE (or any similar generic comparison) is the only/best place to look at when comparing frameworks. Perhaps my interpretation is wrong?

Thanks for explaining, I can totally see how you read that comment. Yes I was just talking about performance. And I totally agree with you that a synthetic benchmark is most likely to be valid and useful when it is narrowly scoped and tries to answer just a couple of very specific questions, and not determine “which is faster”. I don’t consider TE successful right now, and I think that is mostly due to the ridiculous scope they have carved out for themselves, and a lack of focus and quality control. Where we differ is I think that this approach can have merit. People will never stop asking “which is faster”, and without TE all you will have is private data that is valid but cannot be shared, and one-sided marketing collateral churned out by vendors and evangelists.

[josevalim]

@brightball

Does activerecord hold a single database connection for every operation on the request while Ecto checks it out and immediately hands it back on a per query basis in case it’s needed elsewhere on the BEAM?

Yes.

I find benchmarks interesting but drawing conclusions from them is very hard. For example, can we conclude that Ecto slower is than Active Record?

• Those benchmarks are comparing Phoenix vs Rails. We would need to consider all of the impact the “web” part of those choices cause on benchmarks. However, since it is expected that Phoenix is faster than Rails, and the other techempower tests seem to imply so, this actually gives more points to ActiveRecord, because AR seems to be faster even when Phoenix as a whole is faster. On the other hand, Phoenix is using Poison for encoding JSON and faster alternatives exist, and the encoding to JSON is definitely part of what is being measured here.

• On the other hand, applications for both frameworks have been heavily modified and stripped. It is really hard to know if they are doing the same work. For example, both frameworks set the logger level to :error which is a no-no for most production deployments. The Rails benchmarks went as far as to hardcode and freeze important headers, such as Date and Server. No application should ever return the same Date header for ever and ever. One may think this does not matter but it is worth noting that Cowboy uses an ETS table to cache date lookups, or at least used to, because simply looking up dates and converting them to strings shows up on such benchmarks.

• And then there is the point that @brightball raised. Ecto checks out the connection from a pool on every query, Rails uses one connection per request (disclaimer: I am not 100% sure this is true on recent versions though). The Ecto approach leads to better resource utilization on mixed CPU and IO workflows because you are not holding the connection when you are not using it but in those cases it means you have the pool lookup overhead one every query. Case in point: wrapping Ecto lookups in a transaction, as to force it to use a single connection, makes it about 50% faster. From certain perspectives, this can even be considered an Ecto limitation. For example, if you are running scripts that perform many operation against the database, do you really want to check out the connection every time? Probably not.

As you can see, it is really hard to answer which one is faster. Based on the data above, I would conclude that Active Record is faster for this particular workload but I wouldn’t bet on Ecto nor Active Record. It is a 2x difference and minor changes can offset those.

[brightball]

Just curious, but would Ecto.Multi work around that?

[josevalim]

Yes, but only because you execute the multi inside Repo.transaction. So it wouldn’t be any different from calling Repo.transaction with an anonymous function yourself.

[michalmuskala]

It’s possible that with the current work in db_connection that redesigns transaction handling, we will be able to provide a way for checking out a connection for longer than a single query without a transaction.

[nsuchy]

@OvermindDL1

It’s pretty well proven nowadays that techempower is a joke, the lack of decent code across near all of their tests is just reckless and shows that they are not to be taken seriously.

I’ve never heard of techempower until today. I’ll keep that in mind.

[svsdehh]

@michalmuskala

I guess this is relevant: https://twitter.com/ivarvong/status/948677921358647296

Actually I do not think, that this is relevant, as a 95 percentile does not say enough about your latency. There can still be 5% of all requests,that are dead slow.

[gon782]

@svsdehh

Actually I do not think, that this is relevant, as a 95 percentile does not say enough about your latency. There can still be 5% of all requests,that are dead slow.

Generally speaking I’d agree, but with a language optimized for latency (not allowing long-running requests to clog the system & no stop-the-world GC) it’s unlikely that anything short of a mis-design or just very long-running requests by nature would give you a much higher average latency.

[svsdehh]

@gon782

it’s unlikely that anything short of a mis-design or just very long-running requests by nature would give you a much higher average latency.

may I quote cmkarlsson:

@cmkarlsson

Even under 95% CPU load we still manage to have the maximum latency within reasonable numbers. ( average 15ms, max 200 ms)

his max numbers are more than 10 times bigger than his average. So, thats not that unlikely to happen.