I haven’t yet. Appreciate those suggestions.
scouten
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
To be clear, I was making two distinct points.
- Benchee
ipscount only a single process, not N processes where N is the parallelism count. - Benchee parallelism may run in batches which would limit throughput.
#1 is the most important, and isn’t addressed by your example as far as I can tell. There is strong evidence for #1, since both the example you have and the example from the docs shows literally < 0 speed up from parallelism. It’s also trivially easy to prove:
Benchee.run(
%{
"sleep" => fn ->
Process.sleep(1_000)
end
},
parallel: 10,
time: 10
)
warmup: 2 s
time: 10 s
memory time: 0 ns
parallel: 10
inputs: none specified
Estimated total run time: 12 s
Benchmarking sleep...
Name ips average deviation median 99th %
sleep 1.00 1.00 s ±0.02% 1.00 s 1.00 s
If parallelism counted towards IPs, then IPs shouldn’t be 1.0. This is a serious methodological problem, which is great!
#2 could be false, I agree that the docs are unclear. I’m going to see if I can test this more directly.
3
kwando
So I tried your benchmarks on my computer, MacBook Pro 2019 with Postgres.app
If I look in the database after both test runs I get the following results:
ecto: 116378 items
pgbench: 190140 items (~1.63x)
Is this inline with your results?
edit: I sat the benchee :warmup option to 0
0
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
This is critical, and I think affirms the point #1 I was making before. Benchee is running multiples of the benchmark, but only counting a single thread for the IPs. The total in the database creates a clearer picture.
When using mix however warmup is critical, since mix lazy loads code. The first second or so will have a significant performance penalty.
0
kwando
Yes, I could not figure out how to reset the database between the warmup and the test run… but that means it can only get better 
1
scouten
Ah, these are great points. I have some personal medical appointments (routine stuff, no big deal) that are going to require me to step away from the computer for the next few hours. Other members of my team are watching this thread and may try some of these suggestions and I may try once I’m back online. Looking forward to hearing what is found.
Thanks, especially, for the observation in #1: That … surprised me. (Side note: Do you have a different benchmarking tool that you prefer?)
0
scouten
I have been in the habit of using mix ecto.drop && mix ecto.create && mix ecto.migrate between each test run.
0
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
Doing a fixed numer of inserts (~100) before the actual benchmark would probably work as a warmup, and then we could run the benchmark without warmups. Then, just subtract that number from the total in the database.
2
scouten
Filed an issue on the benchee repository about this. I agree that the ips is misleading in this case.
0
OvermindDL1
Instruction Per Second (IPS) is the measure of how many times the callback can be called in one second per benchmarking scaffold.
Parallel means that multiple benchmarking scaffolds can be run simultaneously, each are still distinct.
If you want to test parallelism then you set Benchee to no parallelism (1) and you create the parallelism inside your test structure. This is pretty universal among all benchmarkers in all languages.
Setup the connection in setup, benchmark parallel connections within the benchmark (an input is great for this so you can test different concurrency levels). Do not have Benchee run in parallel. In general never every run Benchee in parallel except for memory testing.
Precisely, because Benchee’s Parallelism is for running tests faster but more inaccurately, it is not to ‘test’ parallelism.
I prefer to setup load in the setup call, much more control. ^.^
Read above in my post here.
Both correct.
You don’t, just absolutely don’t run Benchee in parallel mode for accurate timing measurements, it’s for other purposes.
Not misleading, it is reporting precisely what it should be reporting. If it reported anything else, like 10, it would be absolutely outright wrong.
This is the absolute right answer.
Given that Ecto is doing type conversions, pool handling, etc… This seems quite accurate.
6