Best way to keep read-only table in memory

Will do. I’ll try to implement at least two of the ones suggested here, and compare it against the baseline based on structs.

Thanks again

Yes, it’s super simple if you don’t need to add things at runtime.

defmodule Data do
  # load external data
  @data [
    %{id: 1, other_key: :a},
    %{id: 2, other_key: :b},
    %{id: 3, other_key: :c},
  ]


  for key <- [:id, :other_key] do
    for row <- @data do
      def unquote(:"by_#{key}")(unquote(row[key])) do
        unquote(Macro.escape(row))
      end
    end

    def unquote(:"by_#{key}")(_) do
      nil
    end
  end

end
2 Likes

and if I want to get the value of any key, Do I do value = Data.“by_#{key}”?

Sorry, I’m really struggling with macros

In this example, you do Data.by_id(1) or Data.by_other_key(:a).

Or if you want a unified api, you can put the key in argument as well:

def by(unquote(key), unquote(row[key])) do

Then call it like Data.by(:id, 1)

1 Like

Got it. Thank you

OK. I hit an issue. I have the data in a CSV file, and to load it, I’m doing this:

  def csv_to_map do
    "../test0.csv"
    |> Path.expand(__DIR__)
    |> File.stream!()
    |> CSV.decode(headers: true)
    |> Enum.to_list()
  end

That works fine, but if now I save it to a variable instead of the attribute you use, I get an error:

data = csv_to_map

for key <- [:id, :other_key] do
  for row <- data do
    def unquote(:"by_#{key}")(unquote(row[key])) do
      unquote(Macro.escape(row))
    end
 end

== Compilation error in file lib/catalog.ex ==
** (CompileError) lib/catalog.ex:15: undefined function key/0
(elixir) src/elixir_bitstring.erl:142: :elixir_bitstring.expand_expr/4
(elixir) src/elixir_bitstring.erl:27: :elixir_bitstring.expand/7
(elixir) src/elixir_bitstring.erl:20: :elixir_bitstring.expand/4
(stdlib) lists.erl:1354: :lists.mapfoldl/3
(elixir) expanding macro: Kernel.def/2

That key variable is defined in the outer for comprehension, so I don’t understand why switching from an attribute to a variable makes a difference.

Thanks for your help

Try this (I know embedded for’s sometimes have issues I’ve seen so it’s best to combine them regardless, like this):

data = csv_to_map

for key <- [:id, :other_key], row <- data do
  def unquote(:"by_#{key}")(unquote(row[key])) do
    unquote(Macro.escape(row))
 end

Thank you, but I still get the same error. The compiler doesn’t see “key” as being defined and expands it as a function. This is the code I have now:

defmodule Catalog do
  def csv_to_map do
    "../test0.csv"
    |> Path.expand(__DIR__)
    |> File.stream!()
    |> CSV.decode(strip_fileds: true, headers: true)
    |> Enum.to_list()

  end

  def map_list_to_mem do
    data = csv_to_map()

    for key <- [:id, :other_key], row <- data do
      def unquote(:"by_#{key}")(unquote(row[key])) do
        unquote(Macro.escape(row))
      end

      def unquote(:"by_#{key}")(_) do
        nil
      end
    end
  end
end

And, I still get:
== Compilation error in file lib/catalog.ex ==
** (CompileError) lib/catalog.ex:15: undefined function key/0
(elixir) src/elixir_bitstring.erl:142: :elixir_bitstring.expand_expr/4
(elixir) src/elixir_bitstring.erl:27: :elixir_bitstring.expand/7
(elixir) src/elixir_bitstring.erl:20: :elixir_bitstring.expand/4
(stdlib) lists.erl:1354: :lists.mapfoldl/3
(elixir) expanding macro: Kernel.def/2

The contents here need to happen in the module body if you’re trying to generate functions. This will all happen at compile time. If compile time isn’t when you want to do this then you’ll want to pick one of the other strategies.

1 Like

I see. Beginners mistake :slight_smile: I don’t know why I didn’t try that before.

BTW, I was able to do the ETS based solution, and it’s working fine. I decided not to bother with Mnesia as I just have one additional index and I don’t think I’ll gain anything else from it.

But I’d like to finish at least one of the macro based solutions. Even if I don’t use it, I’m learning a lot.

Thanks a lot

1 Like

That’s not the code you gave before, earlier it was top level and it works (I tested), what you gave has the expansion happening inside another function call, which will most definitely fail, it needs to be at top level (like in the module definition itself).

Yes, sorry about that. I copied and pasted leaving the second function behind. Ben got the error as well.

Thank you

OK. I’m not even close to finish the little system I’m building, but I thought I could share some thoughts already.

I did some very rudimentary bench marking with a vanilla-type table ( 1 key - 1 value) per entry, configure each solution to perform 10M reads, and measure the throughput. I didn’t measure the load time (and the writes) because as I said before even several minutes is acceptable in my case. All these were done against only one process.

The ETS based solution clocked at: 1,079,098 reads/sec

I thought that was quite impressive until I saw the FastGlobal based results: 128,205,128 reads/sec

That’s crazy fast. And, the simple macro-based solution suggested by Kabie was even faster: 212,765,957 reads/sec

So, you’re absolutely right. If speed is the main factor, macros are the best choice.

However, having to recompile and redeploy every time a value changes in the data is something that will carry a cost. I’ll keep these solutions in my back pocket, and suggest them as an option, but for the moment I’ll continue with ETS. I also played with a larger table and saw the performance decreasing, but I’m sure is still going to be more than enough for my app.

One last thought, at the beginning of this task (and thread), I dismissed CacheX because I have to maintain two indexes in the main table. However, all the solutions considered (except Mnesia) require me to handle the additional index directly. So, I reconsidered CacheX, and I now think it is the right solution for most scenarios. I’m in a unique situation because my data changes between 3 and 6 times a year.

Thanks again to all of you for your help. You guys rock,

Cruz

3 Likes