Hello, I’m struggling to work through memory usage problems when trying to run many queries at once using Task.async + Task.await. I’m relatively new to Elixir + Phoenix, so bear with me.
I need to fire off multiple POST requests to a third party service at once and collect all of the responses. Here’s a simplified version of what I’m doing now.
[
Task.async(fn -> some_func_that_makes_a_post_req(...) end),
Task.async(fn -> some_func_that_makes_a_post_req(...) end),
Task.async(fn -> some_func_that_makes_a_post_req(...) end),
Task.async(fn -> some_func_that_makes_a_post_req(...) end),
...
]
|> Enum.map(&Task.await/1)
|> Enum.reduce(...) # do something with the data returned
I’m using hackney to send the POST requests
:hackney.post(
"some_url",
[
{"Content-Type", "application/json"}
],
Jason.encode!(query_body),
[:with_body]
This seemed to work well, but once I started profiling my application I noticed massive memory spikes. My Phoenix application uses around 60-80 MB of memory at baseline, but when I spawn ~40 Tasks (which each make 1 post request) I see a massive spike up to 150-200 MB, and the processes seem to hang around for a minute or two.
Using the :observer I was able to track down which processes were hogging memory. I can’t upload images to the elixir forum yet, but I’m seeing numerous ssl_gen_statem:init/1 processes consume ~2 MB each. They stick around long after the post requests have completed and each task has been awaited.
Any ideas on what I’m doing wrong? I want to be able to send many requests at once, parse each response & format them independently, and finally merge all of the results into one Map / struct. Is there a better way to send numerous https requests at once without flying too close to the sun & running into OOM issues?
