JSON module benchmarking

Hello. I have benchmarked the JSON module ( JSON — Elixir v1.20.4 ) against a few libraries, including Erlang’s :jiffy. I’ve noticed a pattern that I find a bit hard to explain/understand and would like to hear your thoughts on it. Basically, JSON is ~2 times faster than :jiffy when encoding but :jiffy is ~2 times faster in decoding. This makes it quite hard to choose from one library to the other, given that one usually needs both encoding and decoding functionalities.

This is the benchmarking setup I had:

defmodule Benchmark.JsonLibrariesComparison do
  def run(_) do
    experiment()
  end

  def payload(size_kb \\ 1) do
    :crypto.strong_rand_bytes(size_kb * 1024)
    |> Base.encode64()
    |> binary_part(0, size_kb * 1024)
  end

  def experiment() do
    small_payload = payload(30)
    medium_payload = payload(130)
    large_payload = payload(230)

    encode_inputs = %{
      "small (30kb)" => small_payload,
      "medium (130kb)" => medium_payload,
      "large (230kb)" => large_payload
    }

    decode_inputs =
      Map.new(encode_inputs, fn {label, term} ->
        encoded = JSON.encode!(term)
        {label, encoded}
      end)

    Benchee.run(
      %{
        "native encode" => fn binary -> JSON.encode!(binary) end,
        "jiffy encode" => fn binary ->
          IO.iodata_to_binary(:jiffy.encode(binary))
        end
      },
      inputs: encode_inputs,
      warmup: 3,
      time: 10,
      memory_time: 2,
      formatters: [
        {Benchee.Formatters.Console, extended_statistics: true}
      ]
    )

    Benchee.run(
      %{
        "native decode" => fn binary -> JSON.decode(binary) end,
        "jiffy decode" => fn binary -> :jiffy.decode(binary, [:return_maps]) end
      },
      inputs: decode_inputs,
      warmup: 3,
      time: 10,
      memory_time: 2,
      formatters: [
        {Benchee.Formatters.Console, extended_statistics: true}
      ]
    )
  end
end

And these are the benchmarking results:

Operating System: Linux
CPU Information: AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx
Number of Available Cores: 8
Available memory: 13.59 GB
Elixir 1.19.5
Erlang 28.3.1
JIT enabled: true

Benchmark suite executing with the following configuration:
warmup: 3 s
time: 10 s
memory time: 2 s
reduction time: 0 ns
parallel: 1
inputs: large (230kb), medium (130kb), small (30kb)
Estimated total run time: 1 min 30 s
Excluding outliers: false

Benchmarking jiffy encode with input large (230kb) ...
Benchmarking jiffy encode with input medium (130kb) ...
Benchmarking jiffy encode with input small (30kb) ...
Benchmarking native encode with input large (230kb) ...
Benchmarking native encode with input medium (130kb) ...
Benchmarking native encode with input small (30kb) ...
Calculating statistics...
Formatting results...

With input large (230kb)

Name                    ips        average  deviation         median         99th %
native encode        2.29 K      437.46 μs    ±10.12%      429.31 μs      672.61 μs
jiffy encode         1.42 K      703.19 μs    ±13.24%      676.83 μs     1129.40 μs

Comparison:
native encode        2.29 K
jiffy encode         1.42 K - 1.61x slower +265.73 μs

Extended statistics:

Name                  minimum        maximum    sample size                     mode
native encode       383.78 μs     1270.27 μs        22.66 K                419.61 μs
jiffy encode        640.02 μs     3358.87 μs        14.13 K                673.90 μs

Memory usage statistics:

Name             Memory usage
native encode        0.133 KB
jiffy encode          3.40 KB - 25.59x memory usage +3.27 KB

All measurements for memory usage were the same

With input medium (130kb)

Name                    ips        average  deviation         median         99th %
native encode        4.01 K      249.48 μs    ±15.08%      240.11 μs      426.44 μs
jiffy encode         2.45 K      407.65 μs    ±18.41%      389.29 μs      704.10 μs

Comparison:
native encode        4.01 K
jiffy encode         2.45 K - 1.63x slower +158.17 μs

Extended statistics:

Name                  minimum        maximum    sample size                     mode
native encode       215.32 μs     1215.31 μs        39.50 K                238.58 μs
jiffy encode        365.83 μs     3238.25 μs        24.28 K                375.40 μs

Memory usage statistics:

Name             Memory usage
native encode        0.133 KB
jiffy encode          2.62 KB - 19.71x memory usage +2.48 KB

All measurements for memory usage were the same

With input small (30kb)

Name                    ips        average  deviation         median         99th %
native encode       17.03 K       58.74 μs    ±24.40%       55.66 μs      115.45 μs
jiffy encode        10.35 K       96.66 μs    ±19.96%       90.58 μs      156.51 μs

Comparison:
native encode       17.03 K
jiffy encode        10.35 K - 1.65x slower +37.92 μs

Extended statistics:

Name                  minimum        maximum    sample size                     mode
native encode        46.16 μs     2409.24 μs       160.32 K                 55.31 μs
jiffy encode         83.32 μs     2911.81 μs        99.80 K                 90.09 μs

Memory usage statistics:

Name             Memory usage
native encode        0.133 KB
jiffy encode          1.54 KB - 11.59x memory usage +1.41 KB

All measurements for memory usage were the same
Operating System: Linux
CPU Information: AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx
Number of Available Cores: 8
Available memory: 13.59 GB
Elixir 1.19.5
Erlang 28.3.1
JIT enabled: true

Benchmark suite executing with the following configuration:
warmup: 3 s
time: 10 s
memory time: 2 s
reduction time: 0 ns
parallel: 1
inputs: large (230kb), medium (130kb), small (30kb)
Estimated total run time: 1 min 30 s
Excluding outliers: false

Benchmarking jiffy decode with input large (230kb) ...
Benchmarking jiffy decode with input medium (130kb) ...
Benchmarking jiffy decode with input small (30kb) ...
Benchmarking native decode with input large (230kb) ...
Benchmarking native decode with input medium (130kb) ...
Benchmarking native decode with input small (30kb) ...
Calculating statistics...
Formatting results...

With input large (230kb)

Name                    ips        average  deviation         median         99th %
jiffy decode         4.38 K      228.56 μs    ±14.40%      219.09 μs      385.41 μs
native decode        2.39 K      418.64 μs     ±4.11%      416.74 μs      440.77 μs

Comparison:
jiffy decode         4.38 K
native decode        2.39 K - 1.83x slower +190.08 μs

Extended statistics:

Name                  minimum        maximum    sample size                     mode
jiffy decode        212.25 μs     1402.06 μs        42.96 K                218.95 μs
native decode       376.86 μs     2107.66 μs        23.70 K                413.60 μs

Memory usage statistics:

Name             Memory usage
jiffy decode             64 B
native decode           328 B - 5.13x memory usage +264 B

All measurements for memory usage were the same

With input medium (130kb)

Name                    ips        average  deviation         median         99th %
jiffy decode         7.80 K      128.29 μs     ±5.77%      126.48 μs      144.29 μs
native decode        4.16 K      240.63 μs    ±12.97%      234.46 μs      384.16 μs

Comparison:
jiffy decode         7.80 K
native decode        4.16 K - 1.88x slower +112.35 μs

Extended statistics:

Name                  minimum        maximum    sample size                     mode
jiffy decode        119.99 μs     1095.60 μs        75.82 K                126.41 μs
native decode       211.90 μs      943.48 μs        40.94 K                233.76 μs

Memory usage statistics:

Name             Memory usage
jiffy decode             64 B
native decode           328 B - 5.13x memory usage +264 B

All measurements for memory usage were the same

With input small (30kb)

Name                    ips        average  deviation         median         99th %
jiffy decode        27.23 K       36.72 μs    ±42.34%       36.67 μs       47.14 μs
native decode       17.56 K       56.93 μs    ±19.30%       56.01 μs       70.89 μs

Comparison:
jiffy decode        27.23 K
native decode       17.56 K - 1.55x slower +20.21 μs

Extended statistics:

Name                  minimum        maximum    sample size                     mode
jiffy decode         27.17 μs     3489.05 μs       249.05 K                 36.67 μs
native decode        47.63 μs     2833.73 μs       165.92 K                 55.80 μs

Memory usage statistics:

Name             Memory usage
jiffy decode             64 B
native decode           328 B - 5.13x memory usage +264 B

All measurements for memory usage were the same

Any thoughts much appreciated!