Application hangs on Task.await_many

Questions / Help
, ,
1

Hi, I am trying to compute an expensive routine with Task.async/1 and Task.await_many/2 . My code looks like the following:

line_count = length(large_csv)
schedulers = System.schedulers_online - 1 # try to use all the available schedulers minus the scheduler with the main execution flow (does this make sense?)
chunk_size = trunc(line_count / schedulers)

tasks = Enum.map(0..(schedulers - 1), fn index ->
  csv_chunk = large_csv |> Enum.slice(chunk_size * index, chunk_size)
  task = Task.async(fn -> heavy_processing_func(csv_chunk) end)
end)

IO.puts("tasks created")

results = Task.await_many(tasks) # execution hangs here

IO.puts("tasks finished")

My heavy_processing_func prints out when it’s processing finishes. So the above code outputs:

Tasks created
Task 3 finished: 34472 txs
Task 4 finished: 34472 txs
Task 8 finished: 34472 txs
Task 9 finished: 34472 txs
Task 5 finished: 34472 txs
Task 7 finished: 34472 txs
Task 10 finished: 34472 txs
Task 6 finished: 34472 txs
Task 11 finished: 34472 txs
Task 2 finished: 34472 txs
Task 1 finished: 34472 txs
Task 0 finished: 34478 txs

All the tasks created seem to be finished, however for an unknown reason the application hangs when Task.await_many is called. Any ideas why this is happening?

Thanks!

2

that’s super strange and shouldn’t be happening. At the very least Task.await_many should timeout and crash after 5 seconds the way you have it written currently.

3

Is this code running in a web process? The documentation for Task.await_many mentions that if the timeout (default 5000ms) is exceeded, the process that called await_many will exit - maybe this is related to Cowboy handlers proably exit, not being logged or reported anywhere ?

Not directly relevant to this issue, I suspect, but worth calling out - seeking to the middle of a long list repeatedly is expensive. Consider using Enum.chunk_every/2 instead.

1 Like