Is it a problem to use Enum.to_list in ecto stream?

Questions / Help
,
1

Hi friends, imagine you have a function like this:

  def delete_expire_token() do
    stream =
      from(t in UserToken, where: t.expire_time < ^DateTime.utc_now)
      |> MishkaDatabase.Repo.stream()

    MishkaDatabase.Repo.transaction(fn() ->
      Enum.to_list(stream)
    end)
    |> case do
      {:ok, list} ->
        list
        |> Task.async_stream(&delete(&1.id), max_concurrency: 10)
        |> Stream.run
      error ->
        IO.inspect(error)
    end
  end

At first step you run a stream for lazy load like this MishkaDatabase.Repo.stream() and in transaction side you use Enum.to_list to create a list and pass to Task.async_stream function.

When we are using Enum.to_list I think it is not lazy anymore when we have many records in our database and Enum is going to bind all of them on our ram, Am I right? If yes, what should I do in this block code?

Thank you in advance

2

You should be able to feed the result of Repo.stream directly into Task.async_stream:

 MishkaDatabase.Repo.transaction(fn() ->
      stream
      |> Task.async_stream(&delete(&1.id), max_concurrency: 10)
      |> Stream.run
end)

Which is more memory-efficient (as you pointed out), but also shorter and more readable.

1 Like