qzg

qzg

Why does Registry use set and duplicate bag

This is just a general question. I didn’t want to post an issue on the Elixir GitHub, but why does Registry use the set and duplicate bag datatypes in ETS and not ordered_set. I was reading this article which looks like ordered_set is pretty useful, especially for high-concurrency structures.

I’m interested in using the Registry for a high-concurrency use case so this came up. I’m not complaining about the current performance about Registry, but I don’t think I’ve really pushed it yet either.

Thanks!

Most Liked

ruslandoga

ruslandoga

I’m sure elixir team benchmarked different approaches and picked the fastest one for their use-case.

Below is a naive benchmark for inserts and lookups, I’m not taking into account concurrent reads/writes and each function is benchmarked in a single process (AFAIK). I’m also not starting a table per scheduler (in my case I’d use 8 tables of each type) and :erlang.phash2/2 the key to pick the table to insert into / lookup from.

Personally I mostly use ordered_set for range-readable indexes (since ordered_set is a tree) that point to a record in a bag that actually holds the data. Since Registry doesn’t need to iterate the keys in order, it doesn’t need to use ordered_set.

bench_ets.exs

Mix.install([:benchee])

set = :ets.new(:set, [:set, :public])
ordered_set = :ets.new(:ordered_set, [:ordered_set, :public])
bag = :ets.new(:bag, [:bag, :public])
duplicate_bag = :ets.new(:duplicate_bag, [:duplicate_bag, :public])

Benchee.run(%{
  "set insert" => fn -> :ets.insert(set, {:crypto.strong_rand_bytes(3), <<38, 142, 150, 5, 201, 91, 240, 163, 2, 62>>}) end,
  "ordered_set insert" => fn -> :ets.insert(ordered_set, {:crypto.strong_rand_bytes(3), <<38, 142, 150, 5, 201, 91, 240, 163, 2, 62>>}) end,
  "bag insert" => fn -> :ets.insert(bag, {:crypto.strong_rand_bytes(3), <<38, 142, 150, 5, 201, 91, 240, 163, 2, 62>>}) end,
  "duplicate_bag insert" => fn -> :ets.insert(duplicate_bag, {:crypto.strong_rand_bytes(3), <<38, 142, 150, 5, 201, 91, 240, 163, 2, 62>>}) end,
  "control" => fn -> {:crypto.strong_rand_bytes(3), <<38, 142, 150, 5, 201, 91, 240, 163, 2, 62>>} end
})

ms = [{{:_, :_}, [], [true]}]

IO.puts("Counts:")
IO.inspect([
  set: :ets.select_count(set, ms),
  ordered_set: :ets.select_count(ordered_set, ms),
  bag: :ets.select_count(bag, ms),
  duplicate_bag: :ets.select_count(duplicate_bag, ms)
])

:ets.delete_all_objects(set)
:ets.delete_all_objects(ordered_set)
:ets.delete_all_objects(bag)
:ets.delete_all_objects(duplicate_bag)

for tab <- [set, ordered_set, bag, duplicate_bag] do
  Enum.each(1..100_000, fn i -> :ets.insert(tab, {i, i}) end)
end

Benchee.run(%{
  "set lookup" => fn -> Enum.each(1..100_000, fn i -> :ets.lookup(set, i) end) end,
  "ordered_set lookup" => fn -> Enum.each(1..100_000, fn i -> :ets.lookup(ordered_set, i) end) end,
  "bag lookup" => fn -> Enum.each(1..100_000, fn i -> :ets.lookup(bag, i) end) end,
  "duplicate_bag lookup" => fn -> Enum.each(1..100_000, fn i -> :ets.lookup(duplicate_bag, i) end) end,
  "control" => fn -> Enum.each(1..100_000, fn i -> {i, i} end) end
})
> elixir bench_ets.exs

Operating System: macOS
CPU Information: Apple M1
Number of Available Cores: 8
Available memory: 8 GB
Elixir 1.12.3
Erlang 24.1.7

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 35 s

Benchmarking bag insert...
Benchmarking control...
Benchmarking duplicate_bag insert...
Benchmarking ordered_set insert...
Benchmarking set insert...

Name                           ips        average  deviation         median         99th %
control                  1685.13 K        0.59 μs  ±1066.71%           1 μs           1 μs
bag insert                880.97 K        1.14 μs   ±657.97%           1 μs           2 μs
duplicate_bag insert      844.37 K        1.18 μs   ±480.45%           1 μs           2 μs
set insert                841.37 K        1.19 μs   ±511.06%           1 μs           2 μs
ordered_set insert        477.47 K        2.09 μs   ±463.39%           2 μs           3 μs

Comparison:
control                  1685.13 K
bag insert                880.97 K - 1.91x slower +0.54 μs
duplicate_bag insert      844.37 K - 2.00x slower +0.59 μs
set insert                841.37 K - 2.00x slower +0.60 μs
ordered_set insert        477.47 K - 3.53x slower +1.50 μs

Counts:
[set: 4210616, ordered_set: 2842773, bag: 4343634, duplicate_bag: 4919200]

Operating System: macOS
CPU Information: Apple M1
Number of Available Cores: 8
Available memory: 8 GB
Elixir 1.12.3
Erlang 24.1.7

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 35 s

Benchmarking bag lookup...
Benchmarking control...
Benchmarking duplicate_bag lookup...
Benchmarking ordered_set lookup...
Benchmarking set lookup...

Name                           ips        average  deviation         median         99th %
control                     353.35        2.83 ms     ±3.51%        2.81 ms        3.03 ms
set lookup                  146.83        6.81 ms     ±3.15%        6.77 ms        7.75 ms
duplicate_bag lookup        133.99        7.46 ms     ±1.40%        7.46 ms        7.84 ms
bag lookup                  132.82        7.53 ms    ±12.33%        7.35 ms        9.93 ms
ordered_set lookup           76.62       13.05 ms     ±2.41%       13.02 ms       14.22 ms

Comparison:
control                     353.35
set lookup                  146.83 - 2.41x slower +3.98 ms
duplicate_bag lookup        133.99 - 2.64x slower +4.63 ms
bag lookup                  132.82 - 2.66x slower +4.70 ms
ordered_set lookup           76.62 - 4.61x slower +10.22 ms
ityonemo

ityonemo

I think registry needs bag functionality for it’s use as a PubSub system. My guess about set vs ordered_set is for compatibility with people using older OTPs… It might be interesting to copy/paste the Registry code and do a performance analysis on more recent OTP, it would certainly be possible to make a compile-time choice in Elixir registry to default to one or the other depending on OTP version and if there is a difference it would likely be a welcome PR to elixir STDLIB

Where Next?

Popular in Questions Top

aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
beno
I will often find my self writing things similar to: case some_value do nil -&gt; something() "" -&gt; something() _ -&gt; somethi...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31307 143
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New

We're in Beta

About us Mission Statement