I noticed above it mentions that lookups in ETS are O(1), whereas for Map.get I believe they are O(log N), as per documentation.
Why have this disparity, and in that case, why does Map not use ETS behind-the-scenes?
chrism2671
0
NobbZ
Because ETS ist a mutable store, a map is a “single” value you can change and pass around without needing to fear that another process changes your map.
1
FelisOrion
I know this thread is old, but for anyone landing here later: I built Vettore, a tiny Elixir lib with a Rust backend (Rustler) that does fast dot/cosine over big in-memory sets (millions). You load your vectors once into a named collection and then call similarity_search/4 the dot products run in native code (SIMD), so you avoid the Map.take/Enum overhead entirely. If your data starts as {key, weight} pairs, bucket them into a fixed vocab to make a dense float vector; vettore can handles the rest.
sketch:
db = Vettore.new()
{:ok, _} = Vettore.create_collection(db, "vecs", dim, :dot)
# one-time load
:ok = Vettore.batch(db, "vecs", for {id, vec} <- items do
%Vettore.Embedding{value: id, vector: vec}
end)
# query dot-product (filter > 0.5)
{:ok, hits} = Vettore.similarity_search(db, "vecs", query_vec, limit: 100_000)
Enum.filter(hits, fn {_id, score} -> score > 0.5 end)
1