How to disable protocol consolidation warning from Benchee?


Even though I have already done mix clean --all --deps, I still get the next message from Benchee:

Not all of your protocols have been consolidated. In order to achieve the
best possible accuracy for benchmarks, please ensure protocol
consolidation is enabled in your benchmarking environment.

What do I have to do to remove this message?

I am using

$ elixir --version
Erlang/OTP 26 [erts-14.2.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]

Elixir 1.16.0 (compiled with Erlang/OTP 26)

and have the following line in my mix.exs

{:benchee, "~> 1.3", only: [:dev, :test]}

Also, I am running mix test and calling Benchee from a compiled function called from a test.


Hi there,

sounds like your protocols aren’t consolidated - I’m not 100% sure but it may be that you have protocols defined in .exs files and hence they aren’t consolidated? There is some more information here: Protocol — Elixir v1.16.1

benchee checks all protocols for consolidation: benchee/lib/benchee/system.ex at d1228710390ee9d871a7b31e8e458a4b334c7790 · bencheeorg/benchee · GitHub

Maybe you’ve also disabled it in test (as mentioned in the article there).

It’s hard to tell without a code base to access. Right now that warning message can’t be deactivated as it’s one of our “hard warnings” I’d say. Might be persuaded to add a deactivation for it, but generally speaking all protocols should be consolidated for benchmarking.



if I start an IEx session with MIX_ENV=test iex -S mix and then run some of the code in the Benchee’s link you gave above, I get

iex(1)> path = :code.lib_dir(:elixir, :ebin)
iex(2)> mods = Protocol.extract_protocols([path])
[List.Chars, Inspect, String.Chars, Collectable, Enumerable]
iex(3)>, &Protocol.consolidated?/1)
[false, false, false, false, false]

So, none of the above protocols are consolidated, but they are also not my own protocols.

I suppose there is something more general missing in my setting … :thinking:

@PragTob Ok, maybe I should clone Benchee and see what is bing passed to the function all_protocols_consolidated?/1 to see what is going on …

The link to the Elixir docs shows that usually projects disable protocol consolidation in the test env – have you checked if that’s true for your project?

1 Like

I went to check that, and guess what, I had this in my mix.exs:

consolidate_protocols: Mix.env() != :test

I did not go there before because I don’t remember to have added that myself to the mix.exs file.

Perhaps this is some default added when the project was created some time ago? :man_shrugging:

Thanks a lot people!!


Did the warning go away after you made sure test env consolidates protocols?

YES !!

It consolidated all protocols in all files that get compiled in the test environment. I did not check if it consolidated protocols implemented by modules in .exs files (it is not my use case).

Nice. Glad we could help.