Hi,

The benchee documentation states:

Of course, hooks are not included in the measurements (…) Sadly there is the notable exception of too_fast_functions (…). As we need to measure their repeated invocations to get halfway good measurements before_each and after_each hooks are included there. However, to the best of our knowledge this should only ever happen on Windows (because of the bad run time measurement accuracy).

However, if I simply add a call to Logger.debug(...) to my after_each function the measurements are greatly impacted:

Without the Logger

Name                   ips        average  deviation         median         99th %
insert_agent       40.26 K       24.84 μs    ±36.17%       22.57 μs       49.31 μs
insert_agent       44.80 K       22.32 μs    ±26.08%       21.03 μs       41.00 μs
insert_agent       43.39 K       23.05 μs    ±31.96%       21.38 μs       42.45 μs

With the Logger

insert_agent       17.01 K       58.80 μs   ±353.56%       42.42 μs      292.90 μs
insert_agent       17.32 K       57.75 μs   ±249.72%       42.66 μs      321.68 μs
insert_agent       17.72 K       56.43 μs   ±246.62%       42.46 μs      284.70 μs

(showing 3 repeated executions of each scenario just to show that it is repeatable)

Any idea what I might be missing?

I’m running benchee 1.0.1 and the setup is:

Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
Number of Available Cores: 4
Available memory: 7.68 GB
Elixir 1.12.2
Erlang 24.1.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: default
Estimated total run time: 7 s

Thanks for any clarification.