How can I resetrict benchee to run just 5 iterations of each function

Questions / Help
,
1

I am trying to benchmark API which heavily relies on external APIs and I dont want to consume a lot of external APIs when benchmarking. even if I test one simple function, benchee calls this function 255 times which is odd

2

Probably because Benchee.run/2 takes times option, which is the time in seconds for each job, not the number of iteration.

  • time - the time in seconds for how long each individual benchmarking job should be run for measuring the execution times (run time performance). Defaults to 5.
3

Even if I pass time: 0 as options, it still calls the function 166 times… weird

4

From the Benchee docs:

“benchee runs each of your functions for a given amount of time after an initial warmup, it then measures their run time and optionally memory consumption. It then shows different statistical values like average, standard deviation etc. See features”

I guess you’re seeing the “warm up” still happening.

1 Like
5

When it comes to your original question: you cannot restrict the number of calls because they are determined only by time. Here is the function that is responsible for this part: Benchee.Benchmark.Runner.do_benchmark/4. You could define your own Benchee.Benchmark.Runner module and pass to it benchee (some functions takes it as argument) but I don’t recommend this option.

In my opinion, it’s much better to “mock” external APIs (now it may distort your results because response times may vary) and test only your function, just like this:

defmodule My.Module do
  @module Application.compile_env(:my_app, :service_handler)

  def test() do
    @module.external_call()
    |> parse_result()
  end
end

# Now you have to set service_handler and pass My.Module.test/0 to benchee