How to disable protocol consolidation warning from Benchee?

mguimas

Hello,

Even though I have already done mix clean --all --deps, I still get the next message from Benchee:

Not all of your protocols have been consolidated. In order to achieve the
best possible accuracy for benchmarks, please ensure protocol
consolidation is enabled in your benchmarking environment.

What do I have to do to remove this message?

I am using

$ elixir --version
Erlang/OTP 26 [erts-14.2.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit:ns]

Elixir 1.16.0 (compiled with Erlang/OTP 26)

and have the following line in my mix.exs

{:benchee, "~> 1.3", only: [:dev, :test]}

Also, I am running mix test and calling Benchee from a compiled function called from a test.

Thanks

8 comments

#benchee

0 596 8

2024-02-24 20:20:49 UTC

Showing Posts 1 to 8

Show Best Posts
Show All (oldest first)
Show All (newest first)

PragTob

Hi there,

sounds like your protocols aren’t consolidated - I’m not 100% sure but it may be that you have protocols defined in .exs files and hence they aren’t consolidated? There is some more information here: Protocol — Elixir v1.16.1

benchee checks all protocols for consolidation: benchee/lib/benchee/system.ex at d1228710390ee9d871a7b31e8e458a4b334c7790 · bencheeorg/benchee · GitHub

Maybe you’ve also disabled it in test (as mentioned in the article there).

It’s hard to tell without a code base to access. Right now that warning message can’t be deactivated as it’s one of our “hard warnings” I’d say. Might be persuaded to add a deactivation for it, but generally speaking all protocols should be consolidated for benchmarking.

Post #1

mguimas

Hello,

if I start an IEx session with MIX_ENV=test iex -S mix and then run some of the code in the Benchee’s link you gave above, I get

iex(1)> path = :code.lib_dir(:elixir, :ebin)
~c"/usr/local/lib/elixir/bin/../lib/elixir/ebin"
iex(2)> mods = Protocol.extract_protocols([path])
[List.Chars, Inspect, String.Chars, Collectable, Enumerable]
iex(3)> Enum.map(mods, &Protocol.consolidated?/1)
[false, false, false, false, false]

So, none of the above protocols are consolidated, but they are also not my own protocols.

I suppose there is something more general missing in my setting …

Post #2

mguimas

@PragTob Ok, maybe I should clone Benchee and see what is bing passed to the function all_protocols_consolidated?/1 to see what is going on …

Post #3

dimitarvp

The link to the Elixir docs shows that usually projects disable protocol consolidation in the test env – have you checked if that’s true for your project?

Post #4

mguimas

I went to check that, and guess what, I had this in my mix.exs:

consolidate_protocols: Mix.env() != :test

I did not go there before because I don’t remember to have added that myself to the mix.exs file.

Perhaps this is some default added when the project was created some time ago?

Thanks a lot people!!

Post #5

dimitarvp

Did the warning go away after you made sure test env consolidates protocols?

Post #6

mguimas

YES !!

It consolidated all protocols in all files that get compiled in the test environment. I did not check if it consolidated protocols implemented by modules in .exs files (it is not my use case).

Post #7