What to use in Task module for long running database job

Phoenix Forum Questions / Help
,
1

HI all!
I need some help for long running jobs.
in my controller

def send_message(conn, param) do
  recipients = conn.assigns.recipients
  attrs = %{"message_type" => "sms"}
  #This is request to external api.
  results = Sms.send_sms_message_async(phone_numbers, message, msg_sid, status_callback, account, token)
  # This might be long running ecto job depends on messages' quantity
  confirm_order(conn, recipents, attrs, results)
end

I used Task.Supervisor.async_stream in Sms.send_sms_message_async .
and after sending message, each message should recieve status callback from external api.
But I couldn’t get them all successfully, I think it is because of confirm_order function in below
takes long time. I tried to send 600 messages, and confirm_order function should run 600 times and it make my app slows(I checked a performance from Appsignal it says it took 19 seconds for query.ecto :frowning: )

  1. How can I improve this situation? What function will it be appropriate?
  2. And I want a result from first function (sending message) because I need it in second function(confirm_order), How can I do this while I am using async function?
  3. In Task.Supervisor.async_stream function, what is proper number for max_concurrency?
def confirm_order(conn, recipients, attrs, results) do
    case Sales.confirm_order(recipients, attrs) do
      {:ok, %{id: order_id, user_id: user_id}} ->
        Messenger.create_message_status(results, order_id, user_id)
        # Update Bitly status as Saved
        if is_nil(recipients.bitly_id) do
        else
          bitly = Texting.Bitly.get_bitly_by_id(recipients.bitly_id)
          Bitly.confirm_changeset(bitly) |> Bitly.update()
        end

        conn
        |> put_flash(:info, "Message sent successfully. Your analytics data will be updated shortly.")
        |> redirect(to: dashboard_path(conn, :new))
      {:error, _changeset} ->
        conn
        |> put_flash(:error, "Can't send message!")
        |> redirect(to: checkout_mms_preview_path(conn, :index))
    end
  end
2

The first thing I would suggest is to seperate “conn” (web/api) logic from the “send_message” function. How did recipients end up in the conn assigns? Are you able to pass conn params to an ecto struct? This will also have the added benefit of potential validation.

If sending the SMS messages has the potential to be a long task with failures - does the web response really need to wait? Can you persist something to a database with some form of “pending” status that will get updated once complete?

It also sounds like you may be overloading the system with your async tasks. You may want to consider implementing some form of back pressure. Depending on how important these tasks are - this may mean using an actual job queue library. There are plenty of these around.

3

It is controller function.

um no web doesn’t have to wait for the response…

Yes this code is for that purpose, updating status…

        Messenger.create_message_status(results, order_id, user_id)
        # Update Bitly status as Saved
        if is_nil(recipients.bitly_id) do
        else
          bitly = Texting.Bitly.get_bitly_by_id(recipients.bitly_id)
          Bitly.confirm_changeset(bitly) |> Bitly.update()
        end

any suggestions?

4

I haven’t used it in production yet, but Honeydew looks pretty solid. You can use an mnesia backed queue, or if you use postgres already you can turn any ecto model into a job queue, and it does all the expected things like retries, SIGTERM handling etc.

1 Like