How to avoid race conditions?

Questions / Help
, , , ,
1

Hi all,

I have the following configuration:

  • two apps, an app A (that I do not have control over) and an app B (that I own)
  • when someone orders something via app A, app A throws three events usually in the following order:
  • OrderPlaced
  • BillingAddressUpdated
  • PaymentMethodUpdated
  • the app A throws events over RabbitMQ that are then consumed by app B

This means that first, we create an empty domain model %Order{billing_address: nil, id: nil, payment_method: nil} in app B, and then we update it with the data provided by the events.

The problem is that, we receive BillingAddressUpdated and PaymentMethodUpdated more or less at the same time, so when we fetch the corresponding order from the database, it has neither a billing address nor a payment method.

So the BillingAddressUpdatedEventConsumer and the PaymentMethodUpdatedEventConsumer will both fetch the %Order{billing_address: nil, id: 1234, payment_method: nil} record from the database and hold it in memory, then for example, the BillingAddressUpdatedEventConsumer will add the billing address to the order and save it to the database.

However, since the PaymentMethodUpdatedEventConsumer also had the order without the billing address in memory, this consumer will overwrite the change made by the BillingAddressUpdatedEventConsumerand will save the order with the payment method (but the billing address will be erased).

Hence there are orders without billing address or payment method.

To save an order to the database we use the following function in our OrderRepository:

  def save_order(%Order{} = order) do
    order 
    |> cast(@permitted_params)
    |> insert_or_update(order)
  end

Where the permitted params are all the fields from the struct:

  @permitted_params [
    :billing_address,
    :payment_method
  ]

How to avoid such race conditions?

By updating the code so that each consumers only update the fields they are responsible for (i.e. PaymentMethodUpdatedEventConsumer can only update the payment_method field of the order domain model)?

By locking the orders database table? :thinking:

Other suggestions?

2

Easiest thing is to use one consumer for both events, so the updates are serialized?

2 Likes
3

In general I don’t like solving this at the DB layer, but you could look at insert_all

https://hexdocs.pm/ecto/Ecto.Repo.html#c:insert_all/3

It won’t require you to preload the document, and you can do updates using the on_conflict parameters.

It won’t update any auto-generated fields like updated_at, etc.

1 Like
4

Thanks all, I will explore those solutions!

5

This specific pattern is what optimistic locking is designed to guard against - the first update will bump the lock_version column on the order, which will cause the second interaction to roll back with an Ecto.StaleEntryError. You’d need to write code to handle that error by reloading the order and retrying.

4 Likes
6

Thank you @al2o3cr! I decided to use Ecto’s optimistic_lock feature; it was pretty straightfoward to set up and the problem is now solved :muscle: