Is your repository now adjusted to show exact results from both experimental setups without the Enum.random call skewing the results?
dimitarvp
0
scouten
scouten
Now that I have that laptop connected to power again, I re-ran the results and have updated the repo with new numbers.
The O(n) on table size issue is gone. There’s still an absolute difference between Ecto and pgbench, but there are leads we can follow here as well.
0
dimitarvp
Impressive progress. Please post the numbers when convenient, this saga is quite interesting and educational.
6
scouten
Thanks, and I will do so. Hopefully this makes life a little easier for the next person who struggles with perf.
6
scouten
I’ve updated the repo with the results of experiments run during the day. We’re up to about 73% of pgbench performance on the update benchmark.
I’ve run fprof against the latest test harness and it looks like we’re now spending about 87% of the time waiting for messages from Postgres, so there’s probably not much more to squeeze out of this. The only thing that stands out for me is about 3% of time spent on logging (even though the test harness is configured not to produce logs).
2
scouten
@PragTob I think benchee and my tool have different design intents. As you put it earlier in this thread:
Benchee is concerned with measuring and providing data about the time it takes individual function calls to execute.
I’m asking a different question with my tool: What is the theoretical maximum write capacity of a system that uses Elixir, Ecto, and Postgres together when under load? I still call it a “benchmarking tool,” but I’m starting to think it might be more accurate to refer to it as a load-generation tool.
IMHO the maximum write throughput of a Postgres system by itself on any given hardware is shown by pgbench by itself. Right now, I’m showing that a fairly rudimentary Elixir + Ecto application sustains about 73% of that theoretical capacity on the same hardware (a 2015-era MacBook Pro). There’s hope that with logging disabled, we might squeeze out a few more percentage points. (If there are more leads beyond that, I’m all ears!)
Armed with this data, I can now say to others on my team that we’re paying x% for logging, x% for checking out connections on each request, x% for Elixir/Ecto itself, etc., and we can have a discussion about whether these are reasonable costs and whether Elixir/Ecto/Postgres is an appropriate tech stack for our intended use case given our expected (hoped for) traffic.
This is obviously a very simple example. Internally, our situation is more complex and we’re seeing other perf problems that are probably rolled up in that complexity. The original intent of this tool was to mimic our PG system and allow me to layer on complexity, looking for that place where we lose performance.
I think I’ve got a “good enough” feel for baseline Ecto/Postgres write performance that I resume that next stage of the experiment (looking for the details of our internal issues).
Thanks, all, for your patience and help this week.
2
dimitarvp
Results from running the tests on two Mac machines:
| CPU (cores/threads) | Test | IPS | Duration |
|---|---|---|---|
| Xeon-2150B, 10/20 | Elixir create | 27610 | 10s |
| Xeon-2150B, 10/20 | pgbench create | 32204 | 10s |
| Xeon-2150B, 10/20 | Elixir update | 41451 | 30s |
| Xeon-2150B, 10/20 | pgbench update | 135406 | 10s |
| i7-4870HQ, 4/8 | Elixir create | 17661 | 10s |
| i7-4870HQ, 4/8 | pgbench create | 29368 | 10s |
| i7-4870HQ, 4/8 | Elixir update | 23242 | 30s |
| i7-4870HQ, 4/8 | pgbench update | 47860 | 10s |
One thing that struck me as odd that out of all 4 tests, 1 of them has a 30s duration (the Elixir update one) and the rest are all at 10s. Why is that?
Another thing I don’t like about the tests is the inconsistency of the inserted data coming from :rand.uniform(). I would create an ETS table or an in-memory list with 100k pre-calculated random values and would reuse that in all tests (all 4, one after another). Not sure how I’d code that yet.
0