Big maps versus small maps performance

Hey @akoutmos, thanks for taking the time to do this. So it is actually true - it is faster to work with bigger maps compared to small ones… who would expect that

You should also try with a “typical” map (which varies based on application), but something like 5–10 keys. Most of my maps would fall somewhere in that range.

The performance of small maps changes a lot depending on how complex the key term is, while large maps depend less on the complexity of the key.

For instance I would imagine you get different results if the keys were small integers.

Thanks for the input, that’s a great test to run! I’ll create an additional set of synthetic benchmarks using atoms and integers (for completeness although I would suspect atoms and integers to perform roughly the same since atoms are represented internally as integers ?).

Like most synthetic benchmarks this was more of a mental exercise to show the cut over between the two representations of maps otp/erts/emulator/beam/erl_map.h at master · erlang/otp · GitHub. For real world accuracy, smaller maps (5-10 keys) would have been more realistic…but without recompiling Erlang locally with those C macros changed I don’t see a way of doing it. There aren’t any run-time flags to control that right? Perhaps that will be my next mental exercise test haha.

What I would be interested in is the writing speed, everything I know about Elixir by now says writing/reading from maps is really fast, but does map size have an impact on writing, and stuff like update_in\3?

Thanks for doing this.

I was able to sneak in a little bit of benchmarking time during my lunch break and have updated the gist Elixir Benchmark of big versus small maps (inspired by https://twitter.com/akoutmos/status/1266034402422853633) · GitHub to have maps with integers and atoms as keys. Super interesting to see how the results kinda flipped when using atoms/integers as keys Below are the results:

Name                                             ips        average  deviation         median         99th %
Binary Really Small Map - first item         39.12 M       25.56 ns   ±424.34%          25 ns          42 ns
Binary Small Map - first item                38.46 M       26.00 ns   ±215.45%          25 ns          60 ns
Binary Large Map - invalid item              29.38 M       34.04 ns    ±51.90%          33 ns          40 ns
Binary Large Map - first item                23.10 M       43.28 ns    ±46.48%          43 ns          53 ns
Binary Large Map - middle item               22.58 M       44.29 ns   ±272.20%          44 ns          52 ns
Binary Really Small Map - middle item        22.25 M       44.95 ns    ±60.56%          44 ns          63 ns
Binary Really Small Map - invalid item       19.79 M       50.52 ns  ±2762.06%          49 ns          94 ns
Binary Large Map - last item                 19.63 M       50.94 ns   ±741.04%          49 ns         104 ns
Binary Really Small Map - last item          12.94 M       77.29 ns    ±27.06%          76 ns          95 ns
Binary Small Map - middle item               12.89 M       77.56 ns    ±28.47%          76 ns          95 ns
Binary Small Map - invalid item               6.16 M      162.40 ns    ±16.29%         161 ns         177 ns
Binary Small Map - last item                  4.70 M      212.92 ns    ±14.58%         211 ns         229 ns

Name                                           ips        average  deviation         median         99th %
Atom Really Small Map - first item        106.48 M        9.39 ns   ±709.95%           8 ns          33 ns
Atom Small Map - first item               100.67 M        9.93 ns   ±417.93%           9 ns          22 ns
Atom Really Small Map - middle item        85.82 M       11.65 ns  ±2709.93%          10 ns          46 ns
Atom Really Small Map - invalid item       85.01 M       11.76 ns  ±1255.81%          11 ns          21 ns
Atom Really Small Map - last item          80.32 M       12.45 ns   ±171.09%          11 ns          30 ns
Atom Small Map - middle item               77.24 M       12.95 ns   ±341.33%          12 ns          43 ns
Atom Small Map - last item                 62.69 M       15.95 ns   ±167.93%          15 ns          51 ns
Atom Large Map - invalid item              53.46 M       18.71 ns   ±103.92%          18 ns          46 ns
Atom Small Map - invalid item              53.07 M       18.84 ns   ±482.30%          18 ns          29 ns
Atom Large Map - last item                 42.74 M       23.39 ns    ±75.83%          22 ns          41 ns
Atom Large Map - middle item               39.52 M       25.30 ns    ±74.97%          24 ns          41 ns
Atom Large Map - first item                39.19 M       25.52 ns    ±70.75%          24 ns          52 ns

Name                                              ips        average  deviation         median         99th %
Integer Really Small Map - first item         80.64 M       12.40 ns   ±132.30%          12 ns          23 ns
Integer Small Map - first item                78.95 M       12.67 ns   ±131.36%          12 ns          24 ns
Integer Really Small Map - middle item        72.38 M       13.82 ns   ±299.39%          13 ns          25 ns
Integer Really Small Map - last item          65.44 M       15.28 ns    ±94.27%          14 ns          26 ns
Integer Really Small Map - invalid item       64.29 M       15.55 ns   ±104.99%          15 ns          24 ns
Integer Small Map - middle item               57.57 M       17.37 ns  ±4032.76%          16 ns          26 ns
Integer Small Map - last item                 45.08 M       22.18 ns   ±497.67%          21 ns          32 ns
Integer Small Map - invalid item              42.96 M       23.28 ns   ±954.08%          22 ns          32 ns
Integer Large Map - invalid item              42.77 M       23.38 ns   ±114.59%          23 ns          32 ns
Integer Large Map - first item                35.13 M       28.47 ns   ±121.23%          28 ns          37 ns
Integer Large Map - middle item               35.03 M       28.54 ns    ±61.64%          28 ns          37 ns
Integer Large Map - last item                 34.12 M       29.31 ns  ±1685.00%          28 ns          63 ns