Paging Mnesia Tables - how to get the next part of data?

Questions / Help
1

Hi, I’m new to Elixir and I’m trying to learn more about elixir by creating a relational database in elixir. I’m using the :mnesia-select/4 function to see if I can page my data in elixir from the Mnesia table. I am using the following

 def paging(match_spec, page_size) do
    {_atom, {data, cont}} =
      :mnesia.transaction(fn ->
        :mnesia.select(MyTable, match_spec, page_size, :none)
      end)

    result = Mnesia.Table.Query.decode_paging_data(data)
   result
 end

The problem I am facing is that every time I run the function, I get the same data. What I want to do is every time I run this function, I get the next part of the data.

1 Like
2

Per the :mnesia.select/4 docs, you’ll notice that there is an unused variable you are getting back: cont.

The cont is to be used as an argument to :mnesia.select/1, if more or all answers are needed.

As you are not using this to “continue” your pagination, you are simply getting the first page every time. Notice that :mnesia.select/1 gives you a new cont, and if you pass that in, you get the next “page”—repeat until it returns :"$end_of_table" instead.

A complete solution would need to work together with two functions: one to start the pagination, and another to continue it. An :ets example:

defmodule Pagination do
  @spec start_pagination(
          :ets.table(),
          :ets.match_spec(),
          non_neg_integer()
        ) :: {[tuple()], :ets.select_continuation()} | nil
  @doc """
  Starts paginating objects returned by a `match_spec` applied to a `table`.

  If the table is empty, returns `nil`.
  Otherwise, returns the first page with at most `per_page` objects,
  and a continuation to get the `next_page/1`.
  """
  def start_pagination(table, match_spec, per_page) do
    case :ets.select(table, match_spec, per_page) do
      {page, continuation} ->
        {page, continuation}

      :"$end_of_table" ->
        nil
    end
  end

  @spec next_page(:ets.select_continuation()) ::
          {[tuple()], :ets.select_continuation()} | nil
  @doc """
  Continues paginating objects from a `continuation` returned by `start_pagination/3`.

  If the table is empty, returns `nil`.
  Otherwise, returns the next page with at most `per_page` objects,
  and a continuation to get the `next_page/1`.
  """
  def next_page(continuation) do
    case :ets.select(continuation) do
      {page, continuation} ->
        {page, continuation}

      :"$end_of_table" ->
        nil
    end
  end
end

This is similar to “cursor-based” pagination in SQL land, with many of the same caveats.

1 Like
3

In my example above, I chose to return {page, continuation} or nil here, as they fit nicely into the Stream APIs. This allows you to build higher-level APIs, and hide the continuation as an accumulator in Stream.resource/3, letting you do things like:

defmodule Pagination.Stream do
  @spec pages(
          :ets.table(),
          :ets.match_spec(),
          non_neg_integer()
        ) :: Enumerable.t()
  @doc """
  Produces a lazy `Stream` of (at most) `page_size`-length lists of objects
  returned by a `match_spec` applied to a `table`.
  """
  def pages(table, match_spec, per_page) do
    not_started = make_ref()

    Stream.unfold(
      not_started,
      fn
        ^not_started -> Pagination.start_pagination(table, match_spec, per_page)
        continuation -> Pagination.next_page(continuation)
      end
    )
  end

  @spec objects(
          :ets.table(),
          :ets.match_spec(),
          non_neg_integer()
        ) :: Enumerable.t()
  @doc """
  Produces a lazy `Stream` of objects returned by a `match_spec` applied to a `table`,
  loaded into memory in `page_size` chunks.
  """
  def objects(table, match_spec, per_page) do
    pages(table, match_spec, per_page) |> Stream.flat_map(&Function.identity/1)
  end
end

This lets you do things like

Pagination.Stream.pages(table, match_spec, page_size) |> Enum.take(2)

to get the first two pages, or

Pagination.Stream.objects(table, match_spec, page_size) |> Enum.take(100)

to get the first 100 objects, regardless of page_size, but using the pagination mechanism under the hood to limit the number of objects loaded out of :ets at any point in time.

2 Likes
4

Thank you @christhekeele, I see my error, It seems rather straightforward from here.

1 Like