Async task debugging / issue isloation

Questions / Help
#1

The code below works if there are few recipients (email.to) but fails (doesn’t send emails) if recipients are say 128.

email.to
      |> Enum.map(fn to_email -> Task.async(fn -> Mailman.deliver(prepare_email(email, to_email, attachment), config) end)  end)
     |> Enum.map(&Task.await(&1))

I’ve 2 questions.

  1. How do I debug async tasks & isolate issues if possible
  2. Is there a better way to re-write the code.

Note: The sync version works fine (so email and the gateway are fine)

 email.to
        |> Enum.map(fn to_email -> Mailman.deliver(prepare_email(email, to_email, attachment), config) end)

Note 2: The attachment size is about 60 KB (Not sure if it makes any difference though)

#2

There is a default timeout of 5 seconds on Task.await/2. This is normally ok … unless the process that is waiting on Task.await/2 exits at that point (because, e.g. that is the end of its processing), since when the calling process exits so does the Task! (At least, by default …)

So, options to fix this:

  • pass a larger number as the timeout parameter … but you will eventually hit this again, certainly
  • use yield/2 instead of await/2 and keep callign yield/2 until the task is fully complete
  • don’t wait on the process at all, just spawn off the function and let it do its thing … if you need confirmation of completion, have it send a message back to a process that can deal with that
  • put this into a GenServer and/or use a command queue to batch these jobs up and truly run them async

The latter is a rather better solution, if a bit more work, for any number of reasons including: it is easy to rate limit the outgoing tasks if needed (made especially easy with GenStage), you can more easily implement retry mechanisms on temporary failures, etc.

2 Likes
#3

Thank you. It makes good sense. For #4 (command queue) are you suggesting something like RabbitMQ?

#4

Elixir provides everything you need. :slight_smile: You can of course use a more complex external solution like RabbitMQ or even just Reddis, but there are libraries like:

… and of course there is GenStage if you want to roll your own

3 Likes
#5

Excellent! Thank you for your inputs