Alphabet Iterator

Questions / Help
,
7

This is my first foray into benchmarking, so these results may be unreliable.

Operating System: macOS"
CPU Information: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.6.1
Erlang 20.2.4

Benchmark suite executing with the following configuration:
warmup: 3 s
time: 7 s
memory time: 0 μs
parallel: 1
inputs: 100K, 1K, 1M
Estimated total run time: 1 min


Benchmarking LettersAndNumbers.list/2 with input 100K...
Benchmarking LettersAndNumbers.list/2 with input 1K...
Benchmarking LettersAndNumbers.list/2 with input 1M...
Benchmarking generate_list/2 with input 100K...
Benchmarking generate_list/2 with input 1K...
Benchmarking generate_list/2 with input 1M...

##### With input 100K #####
Name                               ips        average  deviation         median         99th %
generate_list/2                   1.44      692.72 ms     ±4.56%      682.07 ms      747.01 ms
LettersAndNumbers.list/2          1.43      697.56 ms     ±4.66%      687.10 ms      787.46 ms

Comparison:
generate_list/2                   1.44
LettersAndNumbers.list/2          1.43 - 1.01x slower

##### With input 1K #####
Name                               ips        average  deviation         median         99th %
generate_list/2                 3.31 K      302.43 μs    ±23.01%         285 μs      695.32 μs
LettersAndNumbers.list/2        2.17 K      459.98 μs    ±14.67%         439 μs      750.16 μs

Comparison:
generate_list/2                 3.31 K
LettersAndNumbers.list/2        2.17 K - 1.52x slower

##### With input 1M #####
Name                               ips        average  deviation         median         99th %
generate_list/2                 0.0141       1.18 min     ±0.00%       1.18 min       1.18 min
LettersAndNumbers.list/2        0.0141       1.18 min     ±0.00%       1.18 min       1.18 min

Comparison:
generate_list/2                 0.0141
LettersAndNumbers.list/2        0.0141 - 1.00x slower

Benchmark Code:

defmodule TicketMarket.MutationBenchmarks do
  alias TicketMarket.LettersAndNumbers
  alias TicketMarket.MutateLayout

  inputs = %{
    "1K" => 1_000,
    "100K" => 100_000,
    "1M" => 1_000_000,
  }

  Benchee.run %{
    "LettersAndNumbers.list/2" => fn(count) -> LettersAndNumbers.list(0, count) end,
    "generate_list/2" => fn(count) -> MutateLayout.generate_list("A", count) end
  },
    formatters: [
      # Benchee.Formatters.HTML,
      Benchee.Formatters.Console
    ],
    time: 7,
    warmup: 3,
    inputs: inputs
end

It’s interesting that both approaches get exponentially slower as the count increases. I do like the simplicity of generate_list/2, so that may be the way I end up going. I don’t expect the count to be higher than 1000, so either function would be usable.

If you have any thoughts about the benchmark, let me know.
Thanks again!

1 Like