What would be the “proper” way to implement a simple persistent key/value store?

ETS is not persistent, DETS and Mnesia are. As you said, there are many choices, so your usage pattern and what you feel more comfortable play a big part.

If I were you I will just use Postgres, especially if you are going to need a relational db down the road anyway.

GenServers are great if you don’t need to scale, as they are notorious for being a bottleneck, the best persistent key value store is DETS if you only need it for a single node or persistent_term.

The Erlang docs are pretty comprehensive, especially if you know your constraints.

Thanks. I think you are right, I feel comfortable with postgres so I I’ll stick to that. So now I just need to figure out how to cache the table in memory and trigger a write to db only when the data has been modified.

I see. I saw something about DETS, but assumed it was probably distributed ETS, hence overkill for my needs and still not persistent. I guess I was wrong. Thank you! I’ll look deeper into DETS, this time without prejudice.

This would be a good fit for cachex or nebulex if you don’t want to roll your own solution.

If you’re using Postgres, use Postgres. Otherwise use SQLite. Create a table with binary keys/values and then use ETS as a write-through cache.

Always write to the DB first and then the cache before returning from the write function. Always read from the cache.

def put(key, value) do
  Repo.insert! %Row{key: key, value: value}
  :ets.insert(@table, {key, value})
end

def get(key) do
  case :ets.lookup(@table, key) do
    [{^key, value}] -> value
    [] -> nil
  end
end

Load the rows from the DB into the cache at startup. If you want to use arbitrary terms just encode them with term_to_binary().

Edit: the put() function above is only correct for a single writer, meaning it cannot be used from multiple processes without synchronization. See @Asd 's more thorough answer below for an example that uses a GenServer as a single writer to serialize writes.

Given the number of times “disk” is mentioned, I’m pretty sure it’s “disk ets”. Distributed ets sure would be nice, though.

I’d do this like this if you want it backed by postgres table

defmodule Table do
  @moduledoc "ets table backed by postgres table"

  use GenServer

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end

  def init(_opts) do
    table = :ets.new(:table_name, [:protected, :set, :named_table])
    entries = for %{key: k, value: v} <- Repo.all(Table), do: {k, v}
    :ets.insert(table, entries)
    {:ok, %{table: table}}
  end

  def handle_call({:write, key, value}, _from, %{table: table}) do
    Repo.insert!(%Table{key: key, value: value})
    :ets.insert(table, {key, value})
  end

  def read(key) do
    case :ets.lookup(:table_name, key) do
      [{_, value}] -> {:ok, value}
      _ -> :error
    end
  end

  def write(key, value) do
    GenServer.call(Table, {:write, key, value})
  end
end

But if you’re okay with just a file on disk, I’d consider using a dets

However, I am going to release much more performant persistent LVM KV db in the upcoming months, so I will reply here again once it’s ready

This code is very wrong and buggy.

Consider two processes calling the put function at the same time. If one does put("key", 1) and other does put("key", 2), it is possible that the order of operations would be

Repo.insert! %Row{key: "key", value: 1}
Repo.insert! %Row{key: "key", value: 2}
:ets.insert(@table, {"key", 2})
:ets.insert(@table, {"key", 1}

And postgres database would have 2 written while ets would have 1. Wrapping it into GenServer (or any other locking mechanism) is a must

You are 100% correct, the OP mentioned settings so I assumed a single writer without even thinking about it.