How to disable protocol consolidation warning from Benchee?

Questions / Help
1

Hello,

Even though I have already done mix clean --all --deps, I still get the next message from Benchee:

Not all of your protocols have been consolidated. In order to achieve the
best possible accuracy for benchmarks, please ensure protocol
consolidation is enabled in your benchmarking environment.

What do I have to do to remove this message?

I am using

$ elixir --version
Erlang/OTP 26 [erts-14.2.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]

Elixir 1.16.0 (compiled with Erlang/OTP 26)

and have the following line in my mix.exs

{:benchee, "~> 1.3", only: [:dev, :test]}

Also, I am running mix test and calling Benchee from a compiled function called from a test.

Thanks

2

Hi there,

sounds like your protocols aren’t consolidated - I’m not 100% sure but it may be that you have protocols defined in .exs files and hence they aren’t consolidated? There is some more information here: Protocol — Elixir v1.16.1

benchee checks all protocols for consolidation: benchee/lib/benchee/system.ex at d1228710390ee9d871a7b31e8e458a4b334c7790 · bencheeorg/benchee · GitHub

Maybe you’ve also disabled it in test (as mentioned in the article there).

It’s hard to tell without a code base to access. Right now that warning message can’t be deactivated as it’s one of our “hard warnings” I’d say. Might be persuaded to add a deactivation for it, but generally speaking all protocols should be consolidated for benchmarking.

2 Likes
3

Hello,

if I start an IEx session with MIX_ENV=test iex -S mix and then run some of the code in the Benchee’s link you gave above, I get

iex(1)> path = :code.lib_dir(:elixir, :ebin)
~c"/usr/local/lib/elixir/bin/../lib/elixir/ebin"
iex(2)> mods = Protocol.extract_protocols([path])
[List.Chars, Inspect, String.Chars, Collectable, Enumerable]
iex(3)> Enum.map(mods, &Protocol.consolidated?/1)
[false, false, false, false, false]

So, none of the above protocols are consolidated, but they are also not my own protocols.

I suppose there is something more general missing in my setting … :thinking:

4

@PragTob Ok, maybe I should clone Benchee and see what is bing passed to the function all_protocols_consolidated?/1 to see what is going on …

5

The link to the Elixir docs shows that usually projects disable protocol consolidation in the test env – have you checked if that’s true for your project?

1 Like
6

I went to check that, and guess what, I had this in my mix.exs:

consolidate_protocols: Mix.env() != :test

I did not go there before because I don’t remember to have added that myself to the mix.exs file.

Perhaps this is some default added when the project was created some time ago? :man_shrugging:

Thanks a lot people!!

2 Likes
7

Did the warning go away after you made sure test env consolidates protocols?

8

YES !!

It consolidated all protocols in all files that get compiled in the test environment. I did not check if it consolidated protocols implemented by modules in .exs files (it is not my use case).

9

Nice. Glad we could help.