Benchee & formatters - easy and extensible (micro) benchmarking

Hi everyone,

I didn’t want to “spam” the form with release announcements after my initial post about benchee back in June but I figured with all the changes and new features now might be a good time to write something again! If it’s too much, please tell me

I just released new versions of my benchmarking library benchee along with benchee_csv, also introducing new formatters benchee_json and finally benchee_html to create nice HTML reports, with 4 different graphs that can also be exported as PNG images! And of course, there also is a blog post about all of it.

benchee has come a long way and I’m particularly excited about it supporting running your suite with different inputs as different implementations may behave differently depending on input size or structure. Also I changed the API of the main interface after a short but good discussion in this very forum. And I gotta say it looks way more elixir now

alias Benchee.Formatters.{Console, HTML}
map_fun = fn(i) -> i + 1 end
inputs = %{
  "Small (10 Thousand)"    => Enum.to_list(1..10_000),
  "Middle (100 Thousand)" => Enum.to_list(1..100_000),
  "Big (1 Million)"       => Enum.to_list(1..1_000_000),
}

Benchee.run %{
  "tail-recursive" =>
    fn(list) -> MyMap.map_tco(list, map_fun) end,
  "stdlib map" =>
    fn(list) -> Enum.map(list, map_fun) end,
  "body-recursive" =>
    fn(list) -> MyMap.map_body(list, map_fun) end,
  "tail-rec arg-order" =>
    fn(list) -> MyMap.map_tco_arg_order(list, map_fun) end
}, time: 10, warmup: 10, inputs: inputs,
   formatters: [&Console.output/1, &HTML.output/1],
   html: [file: "bench/output/tco_detailed.html"]

This then produces outputs thanks to the HTML formatter as you can see in this example report or get a preview with this image:

ips1222×849 50.8 KB

So yeah, I hope you like it. Would be great to hear what you like, or even better what you are missing, not liking or bugs so that I can improve and extend benchee and its associated libraries

Thanks!
Tobi

Ooo, awesome! I love benchee and it just keeps getting better! ^.^

It looks great!

I’m very interested in learning about benchmarking, since AFAIK is a subject with a lot of depth into it. Is there something in which you need particular help with contributions?

Thanks, that means a lot

Oh there is a ton of depth to it and I hope the folks and ElixirLive will agree and find it similarly fascinating as I do

I haven’t yet read too many papers in depth about it. There’s tons of room for contributions, I usually keep a good backlog of issues/features that have come to my mind so that I don’t forgot them and that possible contributors have a place to get started.

Off the top of my head a couple of particularly interesting/important ones to my mind:

• reduce the effect of garbage collection/try to avoid it optionally
• measuring memory consumption
• A new formatter that produces graphs directly, luis ferreira has written a library for that I’m on a train right now so can’t go looking for it
• documentation/fixes for using benchee from Erlang would be great - I still need to dust up my Erlang but that’s probably something the Erlangers here could maybe help with
• on the statistics side benchmarking until a confidence value is reached would be great
• benchee_html has a bunch of smaller design fixes and bigger changes to deal with performance problems - open issues

Of course there is much more, new statistics, providing more system information… there’s lots to do

In case you’re interested and from the greater Hamburg (Germany) area, I’ll be at hh.ex tomorrow talking about benchmarking and benchee with the goal to also hack on benchee together to implement some tiny features (some are especially tagged on github for this).

So if you like here is the meetup page

New tiny releases of benchee in 0.7, benchee_html 0.2 and benchee_json 0.2 have made their way to hex yesterday evening

The biggest feature is that the benchee_html report is now properly split up. Other fixes include goodies like relaxing the Poison dependency as well as adjusting some outputs and parallel statistics generation.

More cool stuff to come of course also benchmarking talks will be given:

• Erlang User Conference, Stockholm, June
• Polyconf, Paris, July

In case anyone wants t hang out and hack

I used Benchee couple times, and it’s definitely a solid solution - probably my “go to” one for all my benchmarking needs.

That said, one thing that bothers me each time I look at the README is that the example benchmark is not inside a module. Code that is not inside modules, is not compiled, but interpreted. This gives vastly different performance characteristics and makes benchmarks pretty much useless.
This is not a huge problem in the example, since the functions immediately call a module (so only the initial, anonymous function call is interpreted), but can lead to false results with more complex things.

Hey! Thanks for the input.

Yes, definitely - you should call to functions that are defined in modules somewhere and compiled and that’s how I do it, maybe it should be made clearer. E.g. I usually just have my benchmarks call functions I have defined elsewhere or a couple of ecto functions

Still, I’d like for the whole suite to be properly compiled as being close to production systems is super important and I can see how people would create erroneous benchmarks through this.

Still - I’d like to know about Elixir works internally and the BEAM VM, but information seems to be really sparse (I read the ELI5 for BEAM and soem usual gotchas but it’s… not much). Pointers are very welcome.

My understanding is that .exs files are interpreted while the .ex are compiled - correct?

The only way, that jumps to mind given my knowledge is correct, is that I’d have people define their code in a module in a .ex and then have a script or some executable call it. Something like:

defmodule MyBenchmark do
  def benchmark do
    Benchee.run(...)
  end
end

And then in a script file just do:

MyBenchmark.benchmark()

Is that the only way? Is there a better way? Your input and or pointers would be highly appreciated @michalmuskala (+ of course everyone else!)

Modules are compiled, no matter if in .exs or .ex file - the only difference is if there’s a compilation artefact (in the form of a .beam file produced). Code outside modules is not compiled.

When it comes to some reference on BEAM internals, I guess the newly released “BEAM Book” is the best comprehensive source out there The BEAM Book: Understanding the Erlang Runtime System - it’s not complete, but it’s an awesome resource nonetheless.

Thanks a ton! Will try to get some of that information into the README of benchee and or the wiki

Also thanks for the link to the book - I had missed that until now. Looking forward to carving out the time to read it!