I am trying to benchmark API which heavily relies on external APIs and I dont want to consume a lot of external APIs when benchmarking. even if I test one simple function, benchee calls this function 255 times which is odd
times option, which is the time in seconds for each job, not the number of iteration.
time- the time in seconds for how long each individual benchmarking job should be run for measuring the execution times (run time performance). Defaults to 5.
Even if I pass
time: 0 as options, it still calls the function 166 times… weird
From the Benchee docs:
“benchee runs each of your functions for a given amount of time after an initial warmup, it then measures their run time and optionally memory consumption. It then shows different statistical values like average, standard deviation etc. See features”
I guess you’re seeing the “warm up” still happening.
When it comes to your original question: you cannot restrict the number of calls because they are determined only by time. Here is the function that is responsible for this part:
Benchee.Benchmark.Runner.do_benchmark/4. You could define your own
Benchee.Benchmark.Runner module and pass to it
benchee (some functions takes it as argument) but I don’t recommend this option.
In my opinion, it’s much better to “mock” external APIs (now it may distort your results because response times may vary) and test only your function, just like this:
defmodule My.Module do @module Application.compile_env(:my_app, :service_handler) def test() do @module.external_call() |> parse_result() end end # Now you have to set service_handler and pass My.Module.test/0 to benchee