Handle multiple atomically operations with side-effects

sezaru

Hello,

Sometimes in my system, I want to change something in the database and also do some other side-effect, for example, filling a cache or sending a message, etc.

Normally what I do is something like this:

alias Ecto.Multi

Multi.new()
|> Multi.insert(:insert_data_to_db, data)
|> Multi.run(:update_cache, fn _, _ -> :ok = MyCache.update(data) end)
|> Repo.transaction(timeout: :infinity)

This works great since if my cache update fails for some reason, my insertion will fail too, avoiding situations that I would have successfully inserted my data into the database and failed to update the cache leaving it into an inconsistent state.

Now, let’s say that I want to also send the data to another process that needs to be done atomically too (it can be another thing too, the process is just an example, but it can be anything that needs to be added atomically).

alias Ecto.Multi

Multi.new()
|> Multi.insert(:insert_data_to_db, data)
|> Multi.run(:update_cache, fn _, _ -> :ok = MyCache.update(data) end)
|> Multi.run(:send_data, fn _, _ -> :ok = SendData.send(data) end)
|> Repo.transaction(timeout: :infinity)

Now the approach above doesn’t work anymore since if send data fails, the database will rollback, but the cache will not.

My question is, how do you guys handle this type of situation that you need to do multiple things and it all needs to be in a “apply all or apply nothing if something fails” manner?

Also, this example I used Ecto, but it can be another database too, or even without a database at all.

Another example, I have one situation that I store the last ids of some data I receive in my node, and I also need to send this data to clients via notification. So I first send the notifications then add they ids to the mnesia database, but if the notifications are sent correctly but my mnesia write fails, then again I would have an inconsistent state.

These types of situations seem very common to me, so I’m really wondering if there is a better solution and how you guys handle it.

6 comments

#ecto #mnesia #ecto_multi #transactions

0 1885 6

2020-08-07 14:39:30 UTC

Showing Posts 1 to 6

Show Best Posts
Show All (oldest first)
Show All (newest first)

mindok

The Saga pattern is meant for this I believe. I haven’t used it myself, but you could try something like GitHub - Nebo15/sage: A dependency-free tool to run distributed transactions in Elixir, inspired by Sagas pattern. · GitHub

Edit: a bit of a backgrounder here https://medium.com/nebo-15/introducing-sage-a-sagas-pattern-implementation-in-elixir-3ad499f236f6

Post #1

derek-zhou

I don’t understand, in your first example if the transaction fails, will the cache update be rolled back? How?

Post #2

sezaru

That’s amazing! Basically what I wanted, thanks a lot of the suggestion!

Post #3

sezaru

If I understood the Multi code correctly, multi will create a transaction with the database (postgres in my case), then will execute the first command which is the insert, and then the second one which is the cache, if both are successful, the transaction will simply close without a window to fail.

But maybe I’m wrong and there is a little window that it can fail if the transaction itself fails as you suggested? For example if the database connection is lost in that window, something like that?

Post #4

derek-zhou

I do not know for sure, However I think there is a possibility. The Saga pattern cited by mindok sounds more robust. You need to provide an explicit rollback function for every single operation that you need to rollback in case of failure in other operations.

Post #5

sezaru

Oh yeah, for sure it’s more robust.

But at the same time it still kinda hard to see the best approach for some cases that it’s not obvious to do the rollback.

For example, let’s say one of your operation is to replace the current value of a row(s) inside the database. After you replaced it, you lost the old information, so I’m not sure what would be the best approach here to restore the old data.

Maybe you can first get the old data and keep it in memory so if a rollback is needed you can replace it back with the old data. That would probably work, but maybe there is a better way.

Post #6

— All posts loaded —