How to handle pattern matching inside enumerators

I have a situation in my Phoenix web application where I’m performing multiple INSERT statements with ecto and I would like to roll back my transaction when any of the insert statements fail and have access to the first possible Ecto.Changeset validation error.

I have something like this:

in_transaction(fn ->
  items
  |> Enum.each(fn item ->
    create_item(item)
  end)
end)

Where create_item/1 is nothing more than an ecto call to create an item via Ecto.Changeset so it will return {:ok, item} on a successful case and an error tuple for unsuccessful case and in_transaction/1 is just some boiler-plate to hide Ecto.Multi logic.

My first attempt to solve my problem was to use with statement inside Enum.each/2:

in_transaction(fn ->
  items
  |> Enum.each(fn item ->
    with {:ok, _item} <- create_item(item) do
      _item
    end
  end)
end)

However, this does not seem to have any effect. I also tried this:

in_transaction(fn ->
  items
  |> Enum.each(fn item ->
    {:ok, _item} = create_item(item)
  end)
end)

Which does throw MatchError, but is not something I would like to handle inside my in_transaction/1 function via try/rescue to get the possible %Ecto.Changeset{errors: errors}.

Is there a better way to short-circuit enumeration when something inside of the enumeration does not match or is try/rescue really the only way? It just doesn’t feel Elixir-way.

@jarmo I suppose we have Ecto.Multi for the use case that you’ve mentioned.

https://hexdocs.pm/ecto/Ecto.Multi.html

It will rollback if one of the transaction fails.

Feels like Ecto.Multi.insert_all/5 is what you need.

Thank you for your replies so far. I’m afraid these do not help me here since my question was more generic to Elixir and I just brought Ecto into game to add more context, but it is not (hopefully) a strict requirement in here.

My question is - how is pattern matching done inside Enum functions properly as to not traverse all elements in the list and get back the first non-matching pattern as a result or :ok or something similar when all was fine within all iterations?

Are you looking for something like this? Enum — Elixir v1.14.2

Not sure what you need but looking at your example you’d probably want to collect the results of each insert and then Enum.map or Enum.filter on them.

Ecto-newbie here.

So with Enum.filter or map, you could then check if any of the inserts failed and rollback if it’s all still inside a transaction?

Absolutely not.

For that you use Ecto.Multi which is amazingly well documented.

We here are kind of trying to generalize a non-Ecto solution because OP said they used Ecto only as an example and needed to get familiar with some Elixir patterns (in terms of collections e.g. lists).

Though I believe that he can now see how an example that’s not picked well can confuse future readers. :confused:

1 Like

Is this a misunderstanding of Enum.each vs. Enum.map? Enum.each always returns :ok

It seems that all of this got quite complex really fast due to my obscure example.

I’m trying to make it as specific as possible.

in_transaction/1 is defined in MyApp.Repo:

defmodule MyApp.Repo do
  use Ecto.Repo,
      otp_app: :my_app,
      adapter: Ecto.Adapters.Postgres

  alias Ecto.Multi
  alias MyApp.Repo

  def in_transaction(fun) do
    with {:ok, %{model: result}} <- run_in_transaction(fun) do
      {:ok, result}
    else
      {:error, :model, error, _} ->
        {:error, error}
    end
  end

  defp run_in_transaction(fun) do
    Multi.new
    |> Multi.run(
       :model,
       fn _repo, _ ->
         fun.()
       end
    )
  |> Repo.transaction
  end
end

create_item/1 is pretty standard to insert an Ecto model into the database:

def create_item(attrs) do
  changeset(%Item{}, attrs)
  |> MyApp.Repo.insert  # returns either {:ok, %Item{} = item} or {:error, %Ecto.Changeset{valid?: false, errors: [...]}} under normal circumstances
end

This abstraction hides Ecto.Multi logic into one module and allows to choose to use a transaction between multiple inserts or not by not changing any Ecto.Schema models.

These building blocks allow me to write code like this:

MyApp.Repo.in_transaction(fn ->
  create_item(%{foo: "bar"})
end)

Code above will either commit %Item{} to the database or roll changes back if there are some problems and will return these problems to the in_transaction caller. Of course inserting one entity does not need a separate transaction and code like this makes more sense:

MyApp.Repo.in_transaction(fn ->
  with {:ok, _item1} <- create_item(%{foo: "bar"}),
       {:ok, _item2} <- create_item(%{foo: "baz"}) do
    {:ok, :all_good} # just so that Ecto.Multi would not roll back
  end
end)

Now, if _item2 creation fails then _item1 will be rolled back as well and this is what we want by using database transactions. So far so good.

Or if I don’t care about transactions/rolling back then code like this works too:

  with {:ok, _item1} <- create_item(%{foo: "bar"}),
         {:ok, _item2} <- create_item(%{foo: "baz"}) do
    {:ok, :all_good}
  end

However, if I have an arbitrary amount of items then the original question became apparent.

For example, this doesn’t work:

def create_all_items(items) do
  Enum.each(items, fn item ->
    create_item(item)
  end) # always traverses whole list and even if some item failed to be created then this information will not be passed up to the caller of `in_transaction/1`

  {:ok, :all_good}  # just so that Ecto.Multi would not roll back, but I'm not sure anymore if anything gets committed in case of any errors inside any `create_item/1`.
end

I tried something like this:

def create_all_items(items) do
  Enum.each(items, fn item ->
    with {:ok, item} <- create_item(item) do
      {:ok, item}
    end
  end) # still returns always :ok and traverses whole list even on non-matches?
end

Then I got to this, which kind of works, but looks really verbose and unexpected by Enum.find/2 to have any side-effect via calling create_item/1:

def create_all_items(items) do
  failed_created_item = Enum.find(items, fn item ->
    !match?({:ok, _item}, create_item(item))
  end)
  
  if failed_created_item do
    failed_created_item # most of the time returns {:error, %Ecto.Changeset{valid?: false, errors: [...]}}
  else
    {:ok, items} # just so that Ecto.Multi would not roll back
  end
end

Basically this stops calling create_item/1 as soon as some item failed to be created, e.g. returned something else than {:ok, item} and the validation error will be passed up to the Ecto.Multi and to the caller of in_transaction/1. I guess the suggested Enum.reduce_while would be as weird and not less verbose.

I ended up doing like this:

def create_all_items(items) do
  Enum.each(items, fn item ->
    {:ok, item} = create_item(item) # this will raise MatchError on errors, but MyApp.Repo.in_transaction needs modifying
  end)
  
  {:ok, :all_good} # just so that Ecto.Multi would not roll back
end

However this requires me to modify run_in_transaction/1 to the following:

  defp run_in_transaction(fun) do
    try do
      Multi.new
      |> Multi.run(
           :model,
           fn _repo, _ ->
             fun.()
           end
         )
      |> Repo.transaction
    rescue
      error in MatchError ->
        with %MatchError{term: {:error, %Ecto.Changeset{}} = validation_error} <- error do
          validation_error
        else
          _ ->
            reraise error, __STACKTRACE__
        end
    end
  end

It looks really ugly due to the fact that I need to have extra with statement inside rescue clause to match only Ecto.Changeset errors and to use reraise to raise any other errors. It’s also very error-prone since it will not work as expected when I’m going to write with statement using <- instead of a regular match operator =. It does not feel something that a real Elixir dev would write. Prove me wrong.

PS! This got me to the next more abstract question about quitting Enum iterator early on from any function (each, map, reduce - you call it). For example in Ruby you can do something like this:

items.each do |item|
  break if item.foo
end

Is there some equivalent pattern/way in Elixir?

1 Like

I meant something in the lines of

  Enum.reduce_while(items, :ok, fn item, acc ->
    case create_item(item) do
      {:ok, _item} -> {:cont, acc}
      {:error, changeset} -> {:halt, {:error, changeset})
    end
  end)
end

Though I really like @sasajuric 's abstraction with Repo.transact/1 from Towards Maintainable Elixir: The Core and the Interface | by Saša Jurić | Very Big Things | Medium

1 Like

I feel there is a misunderstanding how Ecto.Multi works. If you want to insert multiple items in a transaction, and rollback everything if one fails - then do it the Multi way.

Ecto.Multi.new()
|> Ecto.Multi.put(:items, items)
|> Ecto.Multi.merge(:insert_items, fn %{items: items} ->
  Enum.reduce(items, Ecto.Multi.new(), fn item, multi ->
    Ecto.Multi.insert(multi, {:item, item.id}, Item.create_changeset(item)
  end)
end)

Or with Multi.insert_all

I disagree. I think Enum.reduce_while wouldn’t be as weird. But this looks like an example where recursion would be particularly elegant:

def create_items([]), do: {:ok, []}

def create_items([item | items]) do
  with {:ok, stored_item} <- create_item(item),
       {:ok, stored_items} <- create_items(items),
       do: {:ok, [stored_item | stored_items]}
end

Alternatively, as has been mentioned, you can repeatedly add records to multi using Multi.insert. Once you submit such multi to a transaction, it will either succeed completely or fail on first error (and return that error as a result).

1 Like

Okay, seems a little bit better indeed than my Enum.map example with an if statement. I will give it some extra thought.

This will not work because each operation in an Ecto.Multi has to have an unique key. You keep using the same key – :model – an arbitrary amount of times.

IMO one of us can just write you a complete example at one point – sadly I don’t have the time (or focus) and I am only pretending to be smart around here. A keyboard warrior. :003:

But still, what you are after is very doable.

This is what I was trying to get rid of in the first place with creating my own function in_transaction/1 and hiding Multi.new and Multi.run in there so that no other code would know anything about Ecto.Multi or if that code is even run within a transaction and I could easily create my items inside a transaction or outside of a transaction and it would be a choice of the caller not the callee. Also, these atoms :items, :insert_items and so on might be useful at times, but most of the time they’re not that important if it doesn’t matter exactly at which point something got rolled back (this is also easily doable with some logging if really necessary). But as I understand you have a nested transaction at play here with two Multi.new calls which I don’t want to use for my use-case either.

Writing code like you wrote here couples your business logic code really tightly to the Ecto.Multi code and if it ever changes or if there’s a need to get rid of Ecto altogether from your codebase then it will be a nightmare. Also, if not needing to create multiple items then you need to write code twice (or use a transaction with a list of one item which might also look confusing). I really dislike this part of Ecto’s API, thus the initial reason of abstracting it away into darker corners.

Please explain how my Multi.new and Multi.run abstraction is a misunderstanding? It seems to work the way I have been expecting so far without the need to introduce Ecto.Multi everywhere into my codebase and your suggested way does not seem a good way to move away into abstractions.

It seems to be working so far. Is it because of the last return value of not being :ok thus everything gets rolled back?

Edit: Can it be that there is one operation from a Ecto.Multi point of view since there is a one function call fun.() and that’s why it works as expected? At least I have not yet managed to prove otherwise - either everything will be committed to the database or none will be committed.

Very likely, yes. But also when I am saying “it will not work” I don’t mean “the DB operation will fail”, I mean it like “you’re going to lose partial success state because it gets overriden due to the same key being reused” because ultimately Ecto.Multi just uses a Map to preserve changes so far under different keys that you specify as a 2nd argument to its functions. When you reuse a key, you’ll lose those changes. Think Map.put; the previous value under the same key is lost.

@dimitarvp can you bring a practical example when this might cause problems in a production system? I understand why things get lost via Map.put, but not sure I understand what you mean in this particular Ecto.Multi context.