Design puzzle when using Ecto.Multi

In my application I need to perform multiple things during transaction:

1. call HTTP and load some data form a microservice,
2. update some DB records,
3. add a new record DB record,
4. send an SMS verification code.

Ecto.Multi suites this purpose very well, but it looks like I’m facing some roadblocks with it. I need someone else’s opinion on my design.

Currently the code for process above looks like this:

# declaration_controller.ex
def create(conn, %{declaration: declaration_attrs} do
  case DeclarationAPI.create(declaration_attrs) do
    {:ok, %{declaration: declaration} = result} ->
      render(conn, "show.json", result)
    {:error, key, reason, _}
      render(conn, "error.json", %{key => reason})
  end
end

# declaration_api.ex
def create_declaration(declaration_attrs) do
  Repo.transaction(new_declaration_request(declaration_attrs))
end

def new_declaration(declaration_attrs) do
  settings = SettingsMicroserviceAPI.get()

  Multi.new
  |> Multi.update_all(:pending_declarations, pending_declarations_query(declaration_attrs), set: [status: "CANCELLED"])
  |> Multi.insert(:declaration, changeset(declaration_attrs, settings))
  |> Multi.run(:verification_code, Verification, :send_verification_code, [declaration_attrs])
end

There’s a problem with this design. Sometimes SettingsMicroserviceAPI.get() HTTP call will not return expected data: it may stumble upon 404 or even time out. In this case transaction has to be rolled back gracefully, and an error saying something along the lines SettingsMicroserviceAPI has timed out
should be returned to user who requested the initial create action.

So I thought: why don’t I add this HTTP call as a Multi step like this?:

Multi.new
|> Multi.run(:fetch_settings, fn _ -> SettingsMicroserviceAPI.get() end)
|> Multi.update_all(:pending_declarations, pending_declarations_query(declaration_attrs), set: [status: "CANCELLED"])
# ...

However in this case there’s no way I can access the settings in the changeset.

So I see two ways to go from here:

1. Keep the microservice call where it is now, but add a validation to the Ecto.Multi chain,
2. Push the microservice call directly to changeset.

I tried both approaches and here’s what learned.

1. Keep the microservice call where it is now, but add a validation to the Ecto.Multi chain

The microservice call will always return {:ok, data} or {:error, :data}. This suites Ecto.Multi.run requirement for function’s return value so my code becomes this:

def new_declaration(declaration_attrs, user_id) do
  settings = SettingsMicroserviceAPI.get()

  Multi.new
  |> Multi.run(:fetch_settings, fn _ -> settings end)
  |> Multi.update_all(:pending_declarations, pending_declarations_query(declaration_attrs), set: [status: "CANCELLED"])
  |> Multi.insert(:declaration, changeset(pending_declaration_attrs, settings))
  |> Multi.run(:verification_code, Verification, :send_verification_code, [declaration_attrs])
end

Now, Ecto.Multi will validate the settings: in case of {:error, _} = settings the transaction will be rolled back, and a nice error will be returned to controller.

There are two problems with this, however:

1. I have to update the signature of changeset from this:

   def changeset(attrs, settings)
     # ...
   end
   

   to this:

   def changeset(attrs, %{:ok, settings})
     # ...
   end
   

   which looks a little ugly.

2. In case of {:error, _} = settings, I need to add another changeset function clause, as otherwise the code will raise a no matched function error.

2. Push the microservice call directly to changeset

I can do this (the actual code was reduced for brevity):

def create_changeset(attrs) do
  %Declaration{}
  |> cast(attrs, @required_fields)
  |> fetch_settings()
end

def fetch_settings(changeset) do
  result = SettingsMicroserviceAPI.get()

  case result do
    {:ok, settings} ->
      put_change(changeset, :_settings, settings)
    {:error, reason}
      add_error(changeset, :_settings, reason)
  end
end

This works, but with two caveats:

1. In case I need to use settings across multi steps, inside verification code step for example, I’d have to rewrite the verification step to something like this:

   # ...
   |> Multi.run(:verification_code, fn multi ->
        settings = multi.declaration.get_change(:_settings)
   
        Verification.send_verification_code(declaration_attrs, settings))
      end)
   

2. I’m not aware of a standard mechanism for storing arbitrary data alongside the changeset, so I have to use a temporal private _settings key.
   This works right now, but there’s no guarantee changeset becomes more strict in future, to disallow fields not described in the schema?

I’m sticking to the second solution right now as it seems to work (at least on the paper;
I’m yet to implement it). But I wonder how everyone handle HTTP interaction with 3rd parties
during transactions?

Maybe this design is inherently wrong, and there’s an “official way” to deal with these kinds
of situations.