DRY way to perform a side effect after insert/update/delete?

Hi,

I want to run side-effects after insert/update/delete of some of the models.
I will do this using some sort of notify mechanism (pubsub, queue, etc) but my question is: how do I send the notification, that some model was inserted/updated/deleted in a DRY fashion?

The schema modules have various chanegeset functions, but I guess I cannot attach the notification to the changeset. This means in the code that does the insert or update, I need to check if it was {:ok, obj} and remember to run the Notify.obj_created(obj) for example. But this is not great because I have to perform this check in every model, every function that writes to db.

I could create a function to do this notification-on-success which i would use like this:

changeset(%Foo{}, params)
|> Repo.insert()
|> Notify.on_success_notify_created()

… but then I can’t use this function with Ecto.Multi - the notifications would be sent for first steps of the Multi, but the transaction could fail on later steps; the db changes would be rolled back - in which case the notifications would notify about non-existing records.

I read that people use the PostgreSQL NOTIFY mechanism and some smart set of TRIGGERs, but I wonder if I could avoid that - I don’t want to maintain some piece of complexity in SQL; I do not want this mechanism to be always on (perhaps i want to run unit tests without running these side-effects).

How do you handle such use case? In Ruby on Rails one could define callbacks for pretty much any moment of record lifecycle. Is there a similar, alternative approach around Ecto?

1 Like

So why don’t you send the notification after running Repo.transaction(multi) then? (In the success clause.)

I doubt that you have such expensive and slow DB operations, such that the notification would come very late. It will come after what, 0.1 to 5 seconds later? Is that an issue?

If I would really like to make pub-sub notification for the system I would look for Telemetry module of Ecto and subbed to the events there, it seems like a perfect place to bridge Ecto and Pubsub if you have enough metadata.

I am a bit concerned about performance, but if the requests submitted asynchronous you should be fine.

1 Like

Hi! My concer is not performance but having to manually call the notification every time I do Repo.insert etc or commit the operation.

For example, say I have Restaurant and Pizza models, I’d like to be able in my web controller to just call:
Pizza.update(name: “margarita”, price: 20) and not have to remember to also call Notify.pizza_updated in that controller.

I could put the notify call inside the Pizza.update function so it always Calla notify, but then I cannot use Pizza.update as part of transaction because it would send the notification before the transaction completes successfully.

So my problem is where to put the side effect / notification in a controller-(context)-model to have it in one place and not all over the place in the code.

Interesting! Although sounds hacky. I’ll explore this.

Yeah, it is.

The problem with queueing systems is ideally it should be done vise versa:

Frontend sinks messages to the queue, many subscribers are processing these messages (that’s where you’re attaching your custom handlers) and then loop closes and events are submitted back to the frontend.

The problem of this approach is in its complexity. It is really hard to get it right and even if you did you’ll have to pay a hefty price to maintain it.

Whenever you have relatively simple web application it is better to either abuse things like internal messaging like telemetry or build a set of macros which you will use to capture the outputs. For me the former looks less painful, to be fair :slight_smile:

If your side-effects are expressible as jobs, Oban can be a good fit - jobs are scheduled by inserting records into the database, so adding them in an Ecto.Multi will respect the transaction boundary and only run them if the whole transaction commits.

4 Likes

I have a command layer that does side effect.

It’s just a small wrapper around contexts, but with notification/domain event.

3 Likes

You didn’t mention what the side effects are that you’re triggering. For context it would be good to know, what exactly are you trying to achieve? Are you triggering the exact same side effect for each database record across all schemas?

I’d suggest exploring two things:

  • Oban jobs as mentioned by @al2o3cr,
  • Assuming you can write a generic notification code, wrap the Repo:
defmodule MyApp.BaseRepo do
  use Ecto.Repo,
    otp_app: #...
    adapter: # ...
end

defmodule MyApp.Repo do
  def insert(changeset, opts \\ []) do
    MyApp.BaseRepo.insert(changset, opts)
    |> nofity()
  end

  defdelegate ...
end
2 Likes

Here’s a different approach you may consider: GitHub - supabase/realtime: Broadcast, Presence, and Postgres Changes via WebSockets

You’d listen to events even if you perform actions directly on the database.

1 Like

sorry for that omission - the side-effects I have in mind are async operations such as sending notification email, or re-configuration of some process supervision trees (eg. user enables some procesing type and we launch the Broadway pipeline for it, user disables it, and we shut it down).

Take a look at prepare_changes/2, which allows you to register a function on your changeset to be executed right before the changeset is inserted/updated/deleted. In that function you can check the changeset’s action field to determine what action is being taken on the changeset.

Your Foo.changeset/2 function thus might look something like this:

def changeset(foo, attrs) do
  foo
  |> cast(attrs, [...])
  # ...
  |> prepare_changes(fn 
    %Ecto.Changeset{action: :insert} ->
      Notify.notify_created()
      changeset
    %Ecto.Changeset{action: :update} ->
      Notify.notify_updated()
      changeset
    _ ->
      changeset
  end)
end
2 Likes

As you’re probably aware the elixir community tends towards a more explicit approach than ruby (I spent many years with ruby before elixir). The idiomatic elixir approach is to create a layer of services/contexts to keep all of your business logic in. Your views then uses this business logic layer for any operations that it performs. Phoenix has some great documentation for how you might use this approach Contexts — Phoenix v1.6.2.

I’m sure there’s ways to mimic the rails approach here but it’s certainly not a common approach in the elixir community. It might be worth taking a beat to consider contexts as an alternative approach.

4 Likes