Will do. I’ll try to implement at least two of the ones suggested here, and compare it against the baseline based on structs.
Thanks again
Will do. I’ll try to implement at least two of the ones suggested here, and compare it against the baseline based on structs.
Thanks again
Yes, it’s super simple if you don’t need to add things at runtime.
defmodule Data do
# load external data
@data [
%{id: 1, other_key: :a},
%{id: 2, other_key: :b},
%{id: 3, other_key: :c},
]
for key <- [:id, :other_key] do
for row <- @data do
def unquote(:"by_#{key}")(unquote(row[key])) do
unquote(Macro.escape(row))
end
end
def unquote(:"by_#{key}")(_) do
nil
end
end
end
and if I want to get the value of any key, Do I do value = Data.“by_#{key}”?
Sorry, I’m really struggling with macros
In this example, you do Data.by_id(1)
or Data.by_other_key(:a)
.
Or if you want a unified api, you can put the key in argument as well:
def by(unquote(key), unquote(row[key])) do
Then call it like Data.by(:id, 1)
Got it. Thank you
OK. I hit an issue. I have the data in a CSV file, and to load it, I’m doing this:
def csv_to_map do
"../test0.csv"
|> Path.expand(__DIR__)
|> File.stream!()
|> CSV.decode(headers: true)
|> Enum.to_list()
end
That works fine, but if now I save it to a variable instead of the attribute you use, I get an error:
data = csv_to_map
for key <- [:id, :other_key] do
for row <- data do
def unquote(:"by_#{key}")(unquote(row[key])) do
unquote(Macro.escape(row))
end
end
== Compilation error in file lib/catalog.ex ==
** (CompileError) lib/catalog.ex:15: undefined function key/0
(elixir) src/elixir_bitstring.erl:142: :elixir_bitstring.expand_expr/4
(elixir) src/elixir_bitstring.erl:27: :elixir_bitstring.expand/7
(elixir) src/elixir_bitstring.erl:20: :elixir_bitstring.expand/4
(stdlib) lists.erl:1354: :lists.mapfoldl/3
(elixir) expanding macro: Kernel.def/2
That key variable is defined in the outer for comprehension, so I don’t understand why switching from an attribute to a variable makes a difference.
Thanks for your help
Try this (I know embedded for’s sometimes have issues I’ve seen so it’s best to combine them regardless, like this):
data = csv_to_map
for key <- [:id, :other_key], row <- data do
def unquote(:"by_#{key}")(unquote(row[key])) do
unquote(Macro.escape(row))
end
Thank you, but I still get the same error. The compiler doesn’t see “key” as being defined and expands it as a function. This is the code I have now:
defmodule Catalog do
def csv_to_map do
"../test0.csv"
|> Path.expand(__DIR__)
|> File.stream!()
|> CSV.decode(strip_fileds: true, headers: true)
|> Enum.to_list()
end
def map_list_to_mem do
data = csv_to_map()
for key <- [:id, :other_key], row <- data do
def unquote(:"by_#{key}")(unquote(row[key])) do
unquote(Macro.escape(row))
end
def unquote(:"by_#{key}")(_) do
nil
end
end
end
end
And, I still get:
== Compilation error in file lib/catalog.ex ==
** (CompileError) lib/catalog.ex:15: undefined function key/0
(elixir) src/elixir_bitstring.erl:142: :elixir_bitstring.expand_expr/4
(elixir) src/elixir_bitstring.erl:27: :elixir_bitstring.expand/7
(elixir) src/elixir_bitstring.erl:20: :elixir_bitstring.expand/4
(stdlib) lists.erl:1354: :lists.mapfoldl/3
(elixir) expanding macro: Kernel.def/2
The contents here need to happen in the module body if you’re trying to generate functions. This will all happen at compile time. If compile time isn’t when you want to do this then you’ll want to pick one of the other strategies.
I see. Beginners mistake I don’t know why I didn’t try that before.
BTW, I was able to do the ETS based solution, and it’s working fine. I decided not to bother with Mnesia as I just have one additional index and I don’t think I’ll gain anything else from it.
But I’d like to finish at least one of the macro based solutions. Even if I don’t use it, I’m learning a lot.
Thanks a lot
That’s not the code you gave before, earlier it was top level and it works (I tested), what you gave has the expansion happening inside another function call, which will most definitely fail, it needs to be at top level (like in the module definition itself).
Yes, sorry about that. I copied and pasted leaving the second function behind. Ben got the error as well.
Thank you
OK. I’m not even close to finish the little system I’m building, but I thought I could share some thoughts already.
I did some very rudimentary bench marking with a vanilla-type table ( 1 key - 1 value) per entry, configure each solution to perform 10M reads, and measure the throughput. I didn’t measure the load time (and the writes) because as I said before even several minutes is acceptable in my case. All these were done against only one process.
The ETS based solution clocked at: 1,079,098 reads/sec
I thought that was quite impressive until I saw the FastGlobal based results: 128,205,128 reads/sec
That’s crazy fast. And, the simple macro-based solution suggested by Kabie was even faster: 212,765,957 reads/sec
So, you’re absolutely right. If speed is the main factor, macros are the best choice.
However, having to recompile and redeploy every time a value changes in the data is something that will carry a cost. I’ll keep these solutions in my back pocket, and suggest them as an option, but for the moment I’ll continue with ETS. I also played with a larger table and saw the performance decreasing, but I’m sure is still going to be more than enough for my app.
One last thought, at the beginning of this task (and thread), I dismissed CacheX because I have to maintain two indexes in the main table. However, all the solutions considered (except Mnesia) require me to handle the additional index directly. So, I reconsidered CacheX, and I now think it is the right solution for most scenarios. I’m in a unique situation because my data changes between 3 and 6 times a year.
Thanks again to all of you for your help. You guys rock,
Cruz