Thanks again for the helpful suggestions. Please allow me to make some clarifications though as I kinda skimped through a few requirements. My very first original design is actually ETS based with the following forma (each row is well, a row in the ETS)t:
{"doc_uuid1", %{"foo" => 0.123, "bar" => 0.777, ...}}
{"doc_uuid2", %{"foz" => 0.223, "zet" => 0.001, ...}}
...
{"doc_uuid1000000", %{"foo" => 0.93, "zet" => 0.1, ...}}
where the ID of the ETS table is the ID of my document and the value is just a plain Map. Now on average the Map only contains about 200-400 KV pairs, as I store it in a “sparse” way, as in if “doc_uuid1” don’t have the words “zet” it won’t even exists in the KV pair. Now my use case is actually, a new document being ingested, say {doc_id, query_vector} = {"doc_uuid1000001", %{"foo" => 0.01, "zot" => 0.1}}, a relatively short one for the sake of the argument, I would like to know out of the 1M documents which one has a high dot product, then I will actually append this to to ETS table as well. But as you well know,
fun = :ets.fun2ms(fn {doc_uuid, vector} when dot_product(query_vector, vector) > 0.5 -> id end)
:ets.select(:stored_vectors, fun)
won’t work. But reading your suggestion, and please correct me if I am wrong, I could potentially store my data in the ETS as such instead:
{{"doc_uuid1", "foo"}, 0.123}
{{"doc_uuid1", "bar"}, 0.777}
{{"doc_uuid2", "foz"}, 0.223}
...
And perhaps I can pull fun2ms retrieving on the entries, like:
query_keys = Map.keys(query_vector)
fun = :ets.fun2ms(fn {{doc_uuid, key}, vector} when key in Map.keys -> {{doc_uuid, key}, vector} end)
:ets.select(:stored_vectors, fun)
Then perhaps I can do a Enum.group_by on the results to compute the final result I want, which is the ID of the documents that have the dot product of > 0.5.
P/S: Off topic, your book if the first book that our company bought when we switched our main stack to Elixir/Erlang and we have all learnt a lot from it.
