Atomicity in mnesia, specifically Pow's MnesiaCache

I’ve been quite busy in the past week creating my own Pow-based session system :slight_smile: I’ve followed the API guide at https://hexdocs.pm/pow/api.html and then modified it quite heavily to suite my own needs, meaning that sessions now have an ID that is stable for the life of the session, separate from the ID of the refresh token. The tokens themselves are created using Phoenix tokens.

One reason for this setup is that I want to make it possible for a user to see all their current sessions, and log out sessions other than the current one. To do so, I’ve extended Pow’s built-in MnesiaCache with some extra methods. Everything works fine, but now that I start to test things, it seems that the operations in it are not atomic. I’ve used transactions in the code I’ve added myself and I can see transactions being used in Pow’s MnesiaCache, so this does not make sense to me.

The following code is my implementation with doctests and the test module. I’ve hacked around the problem by inserting Process.sleep(30) calls, which is obviously not ideal. Removing causes the tests to fail seemingly because things have not yet been inserted or deleted in mnesia.

Does anybody have an idea what the problem might be? Do I misunderstand mnesia transactions?

defmodule MyAppWeb.Sessions.MnesiaSessionStore do
  @moduledoc """
  Pow persistent session store using Mnesia.
  """
  use Pow.Store.Base,
    ttl: Application.get_env(:my_app_web, :auth)[:refresh_token_ttl],
    namespace: "persistent_session"

  alias MyAppWeb.Sessions.Session

  # Mnesia table object format:
  # {table, key, {session_id, session, [ttl: _, namespace: _], timestamp}}

  @session_store Pow.Store.Backend.MnesiaCache
  @store_config [backend: @session_store]

  @doc """
  Get a session by session id.

  ## Examples

      iex> session = sample_session()
      %MyAppWeb.Sessions.Session{created_at: 123, id: "the_session", last_known_ip: "127.0.0.1", refresh_token_id: nil, refreshed_at: 123, user_id: 1}
      iex> put_session(session.id, session)
      :ok
      iex> Process.sleep(30)
      iex> get_session(session.id)
      session
  """
  @spec get_session(binary) :: Session.t() | :not_found
  def get_session(session_id), do: get(@store_config, session_id)

  @doc """
  Delete a session by session id.

  ## Examples

      iex> session = sample_session()
      %MyAppWeb.Sessions.Session{created_at: 123, id: "the_session", last_known_ip: "127.0.0.1", refresh_token_id: nil, refreshed_at: 123, user_id: 1}
      iex> put_session(session.id, session)
      :ok
      iex> Process.sleep(30)
      iex> delete_session(session.id)
      :ok
      iex> Process.sleep(30)
      iex> get_session(session.id)
      :not_found
  """
  @spec delete_session(binary) :: :ok
  def delete_session(session_id), do: delete(@store_config, session_id)

  @doc """
  Create or update a session by session id.

  ## Examples

      iex> session = sample_session()
      %MyAppWeb.Sessions.Session{created_at: 123, id: "the_session", last_known_ip: "127.0.0.1", refresh_token_id: nil, refreshed_at: 123, user_id: 1}
      iex> put_session(session.id, session)
      :ok
      iex> Process.sleep(30)
      iex> get_session(session.id)
      session
  """
  @spec put_session(binary, Session.t()) :: :ok
  def put_session(session_id, session), do: put(@store_config, session_id, session)

  @doc """
  Get all sessions for the user identified by user_id.

  ## Examples

      iex> session = sample_session()
      %MyAppWeb.Sessions.Session{created_at: 123, id: "the_session", last_known_ip: "127.0.0.1", refresh_token_id: nil, refreshed_at: 123, user_id: 1}
      iex> put_session("1", session)
      :ok
      iex> put_session("2", session)
      :ok
      iex> Process.sleep(30)
      iex> get_all_sessions(session.user_id)
      {:ok, [session, session]}
  """
  @spec get_all_sessions(pos_integer) :: {:error, any} | {:ok, [Session.t()]}
  def get_all_sessions(user_id) do
    object_matcher = fn ->
      :mnesia.match_object({@session_store, :_, {:_, %{user_id: user_id}, :_, :_}})
    end

    case :mnesia.transaction(object_matcher) do
      {:atomic, results} -> {:ok, Enum.map(results, fn {_, _, {_, session, _, _}} -> session end)}
      {:aborted, reason} -> {:error, reason}
    end
  end

  @doc """
  Delete all sessions for the user identified by user_id.

  ## Examples

      iex> session = sample_session()
      %MyAppWeb.Sessions.Session{created_at: 123, id: "the_session", last_known_ip: "127.0.0.1", refresh_token_id: nil, refreshed_at: 123, user_id: 1}
      iex> put_session("1", session)
      :ok
      iex> put_session("2", session)
      :ok
      iex> Process.sleep(30)
      iex> get_all_sessions(session.user_id)
      {:ok, [session, session]}
      iex> delete_all_sessions(session.user_id)
      iex> Process.sleep(30)
      iex> get_all_sessions(session.user_id)
      {:ok, []}
  """
  @spec delete_all_sessions(pos_integer) :: :ok | {:error, any}
  def delete_all_sessions(user_id) do
    object_deleter = fn ->
      :mnesia.match_object({@session_store, :_, {:_, %{user_id: user_id}, :_, :_}})
      |> Enum.each(fn {_, _, {id, _, _, _}} -> delete_session(id) end)
    end

    case :mnesia.transaction(object_deleter) do
      {:atomic, _} -> :ok
      {:aborted, reason} -> {:error, reason}
    end
  end
end

And the test module:

defmodule MyAppWeb.Sessions.MnesiaSessionStoreTest do
  use ExUnit.Case

  import MyAppWeb.Sessions.MnesiaSessionStore

  setup do
    case :mnesia.clear_table(Pow.Store.Backend.MnesiaCache) do
      {:atomic, _} -> :ok
      {:aborted, reason} -> {:error, reason}
    end
  end

  def sample_session(),
    do: %MyAppWeb.Sessions.Session{
      id: "the_session",
      user_id: 1,
      last_known_ip: "127.0.0.1",
      created_at: 123,
      refreshed_at: 123
    }

  doctest MyAppWeb.Sessions.MnesiaSessionStore
end

mnesia definitely is atomic in its operations unless using the dirty functions or multiple functions outside a transaction. Do you have a minimal reproducable example we can copy/paste into IEx that shows otherwise? If so that would be a major OTP bug.

1 Like

The doctests can be pasted into iex, that’s why I write them :slight_smile: I seriously doubt that this heralds the discovery of a serious mnesia bug though, I think I’ve just mucked it up somehow.

I’ll try to create an example when I have the time tomorrow. I was hoping someone with Pow / mnesia experience can tell from the code, especially the get_all_sessions/1 and delete_all_sessions/1 functions that don’t involve Pow code but are all my own, what I’m doing wrong.

They can’t, they depend on Pow, which I don’t have. ^.^;

Mnesia comes with OTP, don’t need Pow to test it. Plus no clue where your get/put etc… functions are defined either so can’t see how they work?

Those are Pow methods inserted by use Pow.Store.Base at the top, the code is on github. You can see here for example that the put function uses a sync_transaction. The problem does seem to be with those Pow-functions though, because when I change this (crashes on final doctest line):

  @doc """
  Delete all sessions for the user identified by user_id.

  ## Examples

      iex> session = sample_session()
      %WisbitsWeb.Sessions.Session{created_at: 123, id: "the_session", last_known_ip: "127.0.0.1", refresh_token_id: nil, refreshed_at: 123, user_id: 1}
      iex> put_session("1", session)
      :ok
      iex> put_session("2", session)
      :ok
      iex> Process.sleep(30)
      iex> get_all_sessions(session.user_id)
      {:ok, [session, session]}
      iex> delete_all_sessions(session.user_id)
      iex> get_all_sessions(session.user_id)
      {:ok, []}
  """
  @spec delete_all_sessions(pos_integer) :: :ok | {:error, any}
  def delete_all_sessions(user_id) do
    object_deleter = fn ->
      :mnesia.match_object({@session_store, :_, {:_, %{user_id: user_id}, :_, :_}})
      |> Enum.each(fn {_, _, {id, _, _, _}} -> delete_session(id) end)
    end

    case :mnesia.transaction(object_deleter) do
      {:atomic, _} -> :ok
      {:aborted, reason} -> {:error, reason}
    end
  end

Into this (calling mnesia directly instead of going through Pow MnesiaCache function):

@doc """
  Delete all sessions for the user identified by user_id.

  ## Examples

      iex> session = sample_session()
      %WisbitsWeb.Sessions.Session{created_at: 123, id: "the_session", last_known_ip: "127.0.0.1", refresh_token_id: nil, refreshed_at: 123, user_id: 1}
      iex> put_session("1", session)
      :ok
      iex> put_session("2", session)
      :ok
      iex> Process.sleep(30)
      iex> get_all_sessions(session.user_id)
      {:ok, [session, session]}
      iex> delete_all_sessions(session.user_id)
      iex> get_all_sessions(session.user_id)
      {:ok, []}
  """
  @spec delete_all_sessions(pos_integer) :: :ok | {:error, any}
  def delete_all_sessions(user_id) do
    object_deleter = fn ->
      :mnesia.match_object({@session_store, :_, {:_, %{user_id: user_id}, :_, :_}})
      |> Enum.each(fn {_, key, {_, _, _, _}} -> :mnesia.delete({@session_store, key}) end)
      # |> Enum.each(fn {_, _, {id, _, _, _}} -> delete_session(id) end)
    end

    case :mnesia.transaction(object_deleter) do
      {:atomic, _} -> :ok
      {:aborted, reason} -> {:error, reason}
    end
  end

It works :slight_smile: but I still don’t understand why.

Ah code! ^.^

Hmm, where you call delete_session(id), that’s the part that isn’t working? Where is it defined, it doesn’t seem to be on Pow’s github?

1 Like

I got it :slight_smile: Turns out that I missed something obvious after all. The delete call is here: https://github.com/danschultzer/pow/blob/master/lib/pow/store/backend/mnesia_cache.ex#L112 which is a GenServer cast which of course returns immediately without waiting for the result. So despite the fact that table_delete on line 247 uses a transaction, I can’t rely on MnesiaCache.delete/2 for atomicity / ordering. Thanks for your help @OvermindDL1!

@danschultzer would there be a drawback to going direcly to mnesia, bypassing that GenServer for delete/put/get? I can tell there’s a lot more stuff happening in there with those invalidators. Is there a reason you implemented it using casts? Wouldn’t it make more sense to use calls in this case, given that transactions are used in the casts’ implementations anyway?

2 Likes

It depends on your app. Cast is used because usually we would like put and delete to be async calls (we don’t care about the response so we don’t want to wait for it). The Mnesia transaction is only there to ensure that the whole cluster is updated.

I use :timer.sleep(100) the few places where I do test the async calls. Otherwise I mock the cache module with sync calls. If this is only an issue with testing, and not a problem for production run then I would do either of the above.

Very cool! It sounds exactly what I try to solve with https://github.com/danschultzer/pow/issues/282

If you have any feedback or code suggestions please do let me know on GH :smile: My plan with the fingerprinting is to eventually get to complete session management and lifecycle tracking: https://github.com/danschultzer/pow/issues/122

3 Likes

Thanks for the info! I’ve decided that using the pow methods is fine, the timeout for a few tests is acceptable and I don’t want to mess up the invalidators-code in MnesiaCache.

As I promised in another thread, I am planning to contribute back :slight_smile: I’ll be working on the implementation for another week or two (we want to get it right, test the code properly and use it for multiple projects), and during this time I hope to open a PR or contribute a guide or something along those lines.

1 Like