Btw, thanks for putting the benchmarks up, they led to a batch of improvements on the Enum.random implementation in Elixir master!
josevalim
Creator of Elixir
scouten
scouten
Re: 30s vs 10s. I’m not sure and probably an artifact of tweaking methodology over time. I’m working up a new set of update numbers based on another tweak I came across this afternoon and standardizing on 30s.
I am seeing results akin to yours as I move to beefier hardware. My work laptop is newer and faster than the personal laptop I’ve been using for the results published so far. pgbench outperforms the Ecto benchmark by a wider margin on that machine than on my personal laptop. In preliminary results, we are seeing a similarly larger discrepancy when we move to provisioned hardware on Amazon (EC2 + RDS). (I hope to have more detail to share on these later today or tomorrow.)
One thing I’m also observing on the beefier laptop is that the Ecto benchmark benefits substantially from increased parallelism. Switching from 10 to 40 processes yielded something like a 50% improvement in reported throughput. I’ll be experimenting with that more over the next ~24 hours and will share here.
3
dimitarvp
I’d also advise standardising the output of your scripts to have the exact same format. IMO there’s not much point in stats like “number of connections without the connection establishing”. You’re not doing micro benchmarks on Postgres itself after all; you are testing real throughput between two competing implementations. Of course connection establishment has to be included in all timing stats.
Another thing I’d definitely do is replace the ad-hoc Task.async / Task.await with the much simpler Task.async_stream. But then I am not sure how would that help (or hamper) your measurements.
0
scouten
What would you be hoping to accomplish with that change? Your “but then…” doesn’t exactly make for a compelling argument. 
0
dimitarvp
My recommendation is mostly related to automatic (read: provided by the Elixir stdlib) parallelisation which in my limited testing in previous projects was clearly superior performance-wise – albeit not by huge margins – compared to hand-crafted Task.async + Task.await workflows.
0
scouten
Interesting findings here: Remove string interpolation from Elixir/Ecto results. by scouten · Pull Request #15 · scouten-archive/pghr · GitHub
First, I’m surprised by the change in pgbench results. I changed to 30-second tests instead of 10-second tests for consistency (thank you @dimitarvp for catching that!), but both the 30s and 10s runs wound up with substantially more throughput than I had observed last week on the same hardware. The only thing I can think of was that I had rebooted the laptop over the weekend and maybe some other process that was on the machine was interfering.
Second, I eliminated the string interpolation from the Elixir/Ecto benchmark and that seems to have “helped” by some percentage. This is the primary change in PR 15.
Third, I re-ran the benchmarks and did more experimentation with the number of parallel tasks. The Elixir/Ecto version benefits substantially from increasing the number of clients up to a point. In the example posted, 80 parallel tasks seemed to be the point where performance peaked. (Having that be an inflection point is actually good for us, given expected usage patterns.)
0
dimitarvp
I’ll gladly rerun the tests on my iMac Pro and MBP 2015 but I am not sure if that wouldn’t pollute this thread. Maybe I should post the revised results somewhere else, what do you reckon?
0