Is Ecto slower than Rails ActiveRecords?

ecto
phoenix

#21

That doesn’t surprise me at all, it is very consistent with other benchmarks I’ve seen (such as Riak vs. Cassandra). One thing to watch out for though is the JVM GC tuning. Erlang and Go are both optimized for latency out of the box, while the JVM is optimized for throughput. How many different garbage collectors did you try on the JVM?

My point is that people are going to look at the best available data, even if it is terrible. They are going to draw conclusions, even if the purveyor of that data says you can’t make those conclusions based on the data. Complaining about this and offering anecdotes is - in my opinion - useless. What would be useful is better data.


#22

I guess this is relevant: https://twitter.com/ivarvong/status/948677921358647296

  • 70+ requests per second all day
  • two VMs
  • 95th percentile response time: 17ms
  • elixir 1.5 / phoenix 1.3

And https://twitter.com/ivarvong/status/948689049224609792

It’s the post rendering route, eg https://theoutline.com/post/2839/justin-timberlake-man-of-the-woods-pivot …, so it fetches a few rows from Postgres and renders templates. We don’t cache HTML responses at the CDN, which allows us to do server-side feature flags, A/B testing, etc.

The lowered need for caching is something that I’ve also experienced. It makes everything so much easier.


#23

Probably one :slight_smile: but I’ll have to check with the java team to know for sure. I am not a java expert so had to read up java GC tuning a bit and I am sure there are things that can improve our java results.

Yes :confused: And it is also hard not to look for data you want to see (confirmation bias). If one wants confirmation ActiveRecord is faster it is easy to find the data to support it and of course the other way around is equally true.

Yes, it seems like it :slight_smile: I just can’t stop keep trying it once it a while when these discussions come up. :smiley: I guess I am just hoping that it makes someone think twice about relying too much on TE or other benchmarks.

Would publishing methods and benchmark results of our testing move it from anecdotal to better data? If so perhaps I need to seek approval to do that.


#24

I respectfully disagree. I think that TE is asking a completely wrong question.

In my opinion the question is never which tool is the fastest. Instead, one of the question should be, is it fast enough. Some other important questions might be, can it scale, is it fault-tolerant, how is the ecosystem, what is the operational support, etc.

None of these seem to be in the focus of TE. They run some synthetic benches, give us flashy graphs, and this is supposed to give us a clue about which tool can should we choose. In my personal experience, in many cases the perf differences in such synthetic benches either don’t matter in the grand scheme of things, or can be significantly reduced by with proper algorithmic and technical interventions.

In some cases, squeezing every nanosecond might matter. I personally think that such cases are special, and a team facing such challenge will get a much more informative answer by conducting their own tests against a simulation/approximation of their own use case, instead of looking at TE, or any other generic/synthetic bench.

Therefore, I personally think that TE is fundamentally flawed in its premise. I also don’t have a high opinion of its implementation, but that’s irrelevant :slight_smile:

Maturity, approachability, community, ease of use, flexibility, runtime guarantees, ecosystem, scalability, fault-tolerance support, are some things that come to mind. And, yes, performance is also relevant, and therefore worth looking into, perhaps by making your own tests and seeing if the tech can deliver.


#25

Its a lot of work to do it well, which is why its always either very narrowly scoped, or poorly done. But yes you would need to publish everything required to reproduce your results. The code, configurations, the specification and machine types, test data, methodology etc (preferrably EC2 or other major cloud provider).


#26

I’m sure you didn’t mean it this way, but this honestly comes off as pretty condescending. The topic of this thread is performance, but I wouldn’t be using Elixir in production if I didn’t know its value proposition beyond raw performance - and I actually I think the performance is generally very good and easy to predict.

TE doesn’t have to solve world hunger in order to address just one question from your list. Scoping their evaluation strictly to performance is totally valid, as long as they don’t say you should uncritically accept their results as the only criteria used to make a decision. And yes as I mentioned, its always better to bench your own application in your own environment. But that doesn’t mean all synthetic performance data is automatically useless. In fact I think TE has demonstrated or at least raised awareness of some bugs and regressions, that were later fixed in their respective projects.


#27

In fairness to techempower (no matter what you think of it generally) you are linking to a page that has a big old warning at the top of the page that its preliminary and potentially defective.


#28

I think TE provided me a very good insight on how to tune perf in Erlang and many other platforms just by using runtime switches or better config leveraging combined community effort. Hence nothing wrong with TE as company.

What I irked about is only that particular result I linked. Will be better if you can put analysis why it happened like that in accordance to banner you saw there :wink:


#29

This is the link to the last finished round

you linked earlier to round 15 which is preliminary and not the final benchmarks


#30

I’m sorry you feel this way, as I didn’t mean to sound like that nor question your view on Elixir.

My statement was a general (not Elixir specific) answer to your following question:

I perceive this question as a rhetorical statement which suggests that TE (or any similar generic comparison) is the only/best place to look at when comparing frameworks. Perhaps my interpretation is wrong?

Either way, if we just limit this discussion to performance, I still think that TE is essentially useless. Extrapolating and generalizing from a synthetic bench comparison is very easy, but I think it’s also misleading.

I agree that some information is better than no information. However, my opinion is that TE data is misinformation, because of the way the bench is conducted, and because of the way it is postulated. And in my book, misinformation is worse than no information.

I’m not against synthetic benches conducted in isolation. In fact, I occasionally reach for those in practice, and in my posts, such as here and here.

My problem is with comparison benches, and extrapolated conclusions that arise from those benches. If you take a look at the mentioned posts, I try hard to avoid drawing such conclusions. This is particularly obvious in the second post, where I do an Elixir take on Pusher’s bench of maximum observed latency in Go. Even though I got much better results in Elixir, not once have I compared it to Go. That post is not about Elixir vs Go, it’s about what can we do with Elixir to reduce the tail latency.

This is how I personally think about evaluating performance in general. It’s not about trying to win a battle of numbers, it’s about trying to achieve acceptable property in the system. YMMV of course :slight_smile:


#31

From elixir code, it should be pretty clear why that thing is slow.

They are benchmarking Poison as JSON encoder as well under DB Query benchmark, which I believe we already aware that Poison is not very performant.

One quick fix is to change the JSON encoder.


#32

Thanks for explaining, I can totally see how you read that comment. Yes I was just talking about performance. And I totally agree with you that a synthetic benchmark is most likely to be valid and useful when it is narrowly scoped and tries to answer just a couple of very specific questions, and not determine “which is faster”. I don’t consider TE successful right now, and I think that is mostly due to the ridiculous scope they have carved out for themselves, and a lack of focus and quality control. Where we differ is I think that this approach can have merit. People will never stop asking “which is faster”, and without TE all you will have is private data that is valid but cannot be shared, and one-sided marketing collateral churned out by vendors and evangelists.


#33

Yes. :slight_smile:

I find benchmarks interesting but drawing conclusions from them is very hard. For example, can we conclude that Ecto slower is than Active Record?

  • Those benchmarks are comparing Phoenix vs Rails. We would need to consider all of the impact the “web” part of those choices cause on benchmarks. However, since it is expected that Phoenix is faster than Rails, and the other techempower tests seem to imply so, this actually gives more points to ActiveRecord, because AR seems to be faster even when Phoenix as a whole is faster. On the other hand, Phoenix is using Poison for encoding JSON and faster alternatives exist, and the encoding to JSON is definitely part of what is being measured here.

  • On the other hand, applications for both frameworks have been heavily modified and stripped. It is really hard to know if they are doing the same work. For example, both frameworks set the logger level to :error which is a no-no for most production deployments. The Rails benchmarks went as far as to hardcode and freeze important headers, such as Date and Server. No application should ever return the same Date header for ever and ever. One may think this does not matter but it is worth noting that Cowboy uses an ETS table to cache date lookups, or at least used to, because simply looking up dates and converting them to strings shows up on such benchmarks.

  • And then there is the point that @brightball raised. Ecto checks out the connection from a pool on every query, Rails uses one connection per request (disclaimer: I am not 100% sure this is true on recent versions though). The Ecto approach leads to better resource utilization on mixed CPU and IO workflows because you are not holding the connection when you are not using it but in those cases it means you have the pool lookup overhead one every query. Case in point: wrapping Ecto lookups in a transaction, as to force it to use a single connection, makes it about 50% faster. From certain perspectives, this can even be considered an Ecto limitation. For example, if you are running scripts that perform many operation against the database, do you really want to check out the connection every time? Probably not.

As you can see, it is really hard to answer which one is faster. Based on the data above, I would conclude that Active Record is faster for this particular workload but I wouldn’t bet on Ecto nor Active Record. It is a 2x difference and minor changes can offset those.


#34

Just curious, but would Ecto.Multi work around that?


#35

Yes, but only because you execute the multi inside Repo.transaction. So it wouldn’t be any different from calling Repo.transaction with an anonymous function yourself.


#36

It’s possible that with the current work in db_connection that redesigns transaction handling, we will be able to provide a way for checking out a connection for longer than a single query without a transaction.


#37

I’ve never heard of techempower until today. I’ll keep that in mind.


#38

Actually I do not think, that this is relevant, as a 95 percentile does not say enough about your latency. There can still be 5% of all requests,that are dead slow.


#39

Generally speaking I’d agree, but with a language optimized for latency (not allowing long-running requests to clog the system & no stop-the-world GC) it’s unlikely that anything short of a mis-design or just very long-running requests by nature would give you a much higher average latency.


#40

may I quote cmkarlsson:

his max numbers are more than 10 times bigger than his average. So, thats not that unlikely to happen.