Benchmark Calls from IEx vs Mix Tasks

If I have a mix project with two libraries that I want to comparatively benchmark - is it fair to call each of them one at a time from an IEx shell (started with iex -S mix) and compare their execution times? Is this significantly different than say, making a mix task, and running it? How much of the IEx experience is interpreted rather than compiled?

1 Like

I may be wrong, but I think iex is entirely interpreted.
Do note, that means that defmodule'ing inside iex will interpret the defmodule call, but that really does compile a module and code run ‘inside’ those functions are fast.

1 Like

If you’re trying to benchmark libraries you’ll have much better and more useful results with a benchmarking library like benchee.

2 Likes

In any VM environment benchmarking is very tricky, you never know if you are benchmarking your actual code or the implementation of the VM. You really need to do many invocations and do a statistical analysis. Using :tc.timer in iex can provide very misleading results.env MIX_ENV=prod also effects the timing results.

Benchee , Benchfella and other similar libraries help, but even with those there are some gotchas. If you are simply looking for order of magnitude results, then :tc.timer in iex is probably ok, but I would not trust it beyond that.

2 Likes

Thanks everyone. I’ll definitely try to check out some of the benching libraries to see what they do for this.