I have a strange situation for which I need your thoughts.

My application sends messages to users on request from a different server. For some reason which we’re yet to identify, the server would bombard my application to send same message to users instead of a single request.

I wanted to reduce the problem of annoying customers in case this unfortunate scenario happens since the other team hasn’t figured out why that happens yet.

This is what I came up with:

# Let's check whether message with the same fingerprint exits within the last 5 minutes.
    # If so, send incident reminder to team and log
    msg = :base64.encode("#{message}:#{to}:#{provider}:#{app}") |> MyApp.Bucket.find_message

     cond do
      #If we don't have the message fingerprint or similar message send within last 5 minutes
      msg == nil ->
        #Add new message in the bucket
        # New conversation, save particulars in the database
        MyApp.Bucket.create_message(%MyApp.Bucket{
          id: :base64.encode("#{message}:#{to}:#{provider}:#{app}"),
          provider: provider,
          app: @utilities.get_app_value(app, "name"),
          recipients: to,
          attempts: 1,
          message: message
        })
       
        #Send message
        process_message_sending(message, to, provider, app)

      true ->
        case to_string(msg.message) == message and DateTime.diff(DateTime.from_naive!( NaiveDateTime.utc_now(),"Etc/UTC"), DateTime.from_naive!(msg.updated_at,"Etc/UTC"), :second) < 300 do
          true ->
              "Multiple request to send received for message id #{msg.id}  with #{msg.attempts+1} attempts so far"  |> IO.inspect
          _ ->
            process_message_sending(message, to, provider, app)
        end
        msg
          |> Map.replace!(:attempts, msg.attempts+1)
          |> MyApp.Bucket.update_message()
    end

The idea is to save every outgoing message in a bucket and cross-check with next request to send out message to the same recipient. If similar message has been sent within the last five minutes, it is meant to ignore.

The challenge with this approach is that, if I perform a load test, I’m only around 70% successful at stopping the messages. It looks like there’s a delay in persistence into the Mnesia Bucket wrapped by memento is a tad slow.

Any idea how this can be addressed?

I’ve also thought about rate limiting through nginx but I’ve never done it before and not sure how to successfully use it in conjunction with my code