And 0.13.1 is live for your pleasure - includes fixes to memory measurements that were exposed by @michalmuskala! Fun stuff like you know, heap_size in the garbage collection information isn’t actually the heap size but the size of the young generation apparently. Anyhow, we’re happy if more people try out memory measurements and break them.
PragTob
3
PragTob
And 0.14.0 is finally live for your pleasure! The major feature is drastically increased precision when measuring: from microseconds to nanoseconds! There’s a blog post showing off the most important changes but you can also check out the Changelog with all the changes.
6
PragTob
It has its own thread but I think it’s worth mentioning here as well: benchee_markdown - the first formatter not written by me/benchee maintainers which makes me very happy because that’s what I always wanted. 
Also we’re gearing up for 1.0 (finally!) - stay tuned 
Not too many changes incoming but more the promise of a stable API/data structure until 2.0 which is hopefully a long way off.
As a preparation of this and showcasing that benchee is a project many people contribute to (be it code, reviews, input etc. @devonestes @wasnotrice or “just” ideas, testing and help with a lot of the really hard stuff or the occassional emoji react so one knows something is a good idea @OvermindDL1 @michalmuskala) benchee and related libraries were moved to benchee · GitHub Other good side effect, people other than myself (namely devon and eric so far) can add people manage things etc.
@mods / @AstonJ / ??? could we maybe rename this thread to just be “benchee & formatters - easy and extensible (micro) benchmarking”, because I mean there’s benchee_html, benchee_json we could also name them all in the title but that’d get long and just having benchee/benchee_csv there seems kinda irritating
3
PragTob
So here is what was promised! Benchee 0.99 and 1.0 (along with all plugins) are a reality 
There aren’t many super hardcore new features, except for showing the absolute difference in the average of measurements to help arguments & informed decisions. Otherwise it’s some polish, documentation, house keeping etc.
0.99 introduces a bunch of deprecation warnings, when you eliminate them you should be good to upgrade to 1.0.
I also want to thank everyone here for their input etc 
Happy benchmarking
8
PragTob
PSA for benchee users on Mac doing nano level benchmarks: There’s some weird behaviour in erlang time measurement on MacOS - if you do nanosecond/low microsecond level benchmarks on MacOS this likely affects you and generates wrong results. More information here: MacOS reports it has nano-second precision, but it most often doesn't --> Bad nano benchmark results · Issue #313 · bencheeorg/benchee · GitHub
2
PragTob
Hello friends, long time no see. Enjoy a new Benchee version with a bugfix (finally!), reduction counting and profiler after benchmarking (yay!) as well as some musings on Open Source and why it took so long.
7
mickel8
Membrane Core Team
Hi @PragTob!
Thank you for your awesome work on benchee!
I have a question regarding std dev. When benchmarking a function that serializes some struct into a binary, I am noticing enormous std dev, like 21k%
Operating System: Linux
CPU Information: Intel(R) Core(TM) i5-9600K CPU @ 3.70GHz
Number of Available Cores: 6
Available memory: 15.55 GB
Elixir 1.14.2
Erlang 25.1
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 2 s
reduction time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 11 s
Benchmarking raw_attr.encode ...
Name ips average deviation median 99th %
raw_attr.encode 5.17 M 193.34 ns ±21393.07% 137 ns 162 ns
Extended statistics:
Name minimum maximum sample size mode
raw_attr.encode 132 ns 80013984 ns 8.88 M 136 ns
Memory usage statistics:
Name Memory usage
raw_attr.encode 32 B
**All measurements for memory usage were the same**
Reduction count statistics:
Name Reduction count
raw_attr.encode 1
**All measurements for reduction count were the same**
This basically means that I shouldn’t look at the average result as it might not be reliable. What about adding outlier detection to remove such measurements? Is this something that is planned or welcomed as a contribution?
I am also curious what can be the reason of such a big std dev.
0
PragTob
Heyo,
yes your average is skewed by a lot. I recommend looking at the median and mode which give you a realistic picture of what is happening.
You could also use the HTML formatter - that one gives you graphs that show the distribution to get an even better view.
Essentially what you’re doing is a nano benchmark - most benchmarking tools (and normal measuring tools) don’t even allow that level of precision. For best results the benchmark should be the only thing running on your OS (not even a UI otherwise) but that won’t get rid of it all as well - if your OS decides to not schedule/run the erlang process for even 1ms that’s even already an outlier of like 10_000 x looking at your numbers there and hence you get the big std dev.
I thought there already was an issue for removing outliers as I had been thinking about it a lot, but apparently there is not (just benchmark until confidence value reached which is similar but different: Option to benchmark until confidence value is reached · Issue #9 · bencheeorg/benchee · GitHub )
I should create one, but I’m also unsure what method to use but iirc there was a definition that I used to know and explain in one of my talks but I forgot. I should probably rewatch one of the talks to see how to implement it 
2
PragTob
Created an issue: Built in outlier detection · Issue #382 · bencheeorg/benchee · GitHub
Further thoughts: esp. due to outliers etc. average is often a terrible value but one we’re used to. I’d always look at all values, but also esp. the median. I talk about that a bit here if you wanna know more: https://youtu.be/C4hqcLwxs3A?t=1918
Another reason for the stddev is just the GC running as well/growing the process memory etc.
1
mickel8
Membrane Core Team
Thanks that’s super helpful!
I did a very short research and there is a rust lib called criterion which does outlier detection. However, they explicitly states that even though outliers are detected, they are not dropped from the data. They also describe how they detect outliers here.
EDIT: Looks like you described in the issue almost the same method for detecting outliers as in the criterion 
0