High Memory Usage: firing off many https requests in Task.async calls has huge memory impact

Hello, I’m struggling to work through memory usage problems when trying to run many queries at once using Task.async + Task.await. I’m relatively new to Elixir + Phoenix, so bear with me.

I need to fire off multiple POST requests to a third party service at once and collect all of the responses. Here’s a simplified version of what I’m doing now.

[
  Task.async(fn -> some_func_that_makes_a_post_req(...) end),
  Task.async(fn -> some_func_that_makes_a_post_req(...) end),
  Task.async(fn -> some_func_that_makes_a_post_req(...) end),
  Task.async(fn -> some_func_that_makes_a_post_req(...) end),
  ...
]
|> Enum.map(&Task.await/1)
|> Enum.reduce(...) # do something with the data returned

I’m using hackney to send the POST requests

:hackney.post(
   "some_url",
   [
     {"Content-Type", "application/json"}
   ],
   Jason.encode!(query_body),
   [:with_body]

This seemed to work well, but once I started profiling my application I noticed massive memory spikes. My Phoenix application uses around 60-80 MB of memory at baseline, but when I spawn ~40 Tasks (which each make 1 post request) I see a massive spike up to 150-200 MB, and the processes seem to hang around for a minute or two.

Using the :observer I was able to track down which processes were hogging memory. I can’t upload images to the elixir forum yet, but I’m seeing numerous ssl_gen_statem:init/1 processes consume ~2 MB each. They stick around long after the post requests have completed and each task has been awaited.

Any ideas on what I’m doing wrong? I want to be able to send many requests at once, parse each response & format them independently, and finally merge all of the results into one Map / struct. Is there a better way to send numerous https requests at once without flying too close to the sun & running into OOM issues?