Best way to compute sum of elements in a list

Questions / Help
1

I am trying to get a grip on Task.async, and Task.async_stream. My eventual idea was to write a program which will sum elements of a given list concurrently. The idea was to contrast this summing vs summing a list in a single process/thread.

Is there a reference to such a code?

Sincerely,
Sudarsan.D

2

What’s wrong with the docs of Task.async_stream/3 and Task.async_stream/5? Both include good examples. Are you having trouble following them?

3

Actually what i intend to acheive is to parallelly compute the sum of elements in a list. But with async_stream what happens is that every elements is taken and passed to a function which can do some operation on it. But i dont want it.

Let us say i have a 8 core system and i have a list of 10 Million elements. I think the best way to sum this list of 10 Million elements is to split this 10 Million elements into 8 parts and have each core do the summation of each of the parts. After that we need to just add the results.

Async_stream seem to support a different use case which is : If i have a list of millions of websites and i wanted to fetch some data from them, i can pass this list (which has the websites) to this async_stream and it will go ahead and concurrently fetch the data.

Sincerely,
Sudarsan.D

5

Yep, I noticed. Here’s one naive solution:

# Load your list here
list = Enum.to_list(1..100)

# In real conditions with a huge list you definitely do NOT want to use `length`
count = length(list)

# Calculate chunk size and round it up
chunk_size = ceil(count / System.schedulers_online())

# - Split the list into chunks
# - Sum each list chunk in a separate scheduler (usually a CPU core/thread)
# - Collect all sums in a list in the shape of: `[ {:ok, sum1}, {:ok, sum2}, ... ]`
# - Only take the sums and discard the `:ok` atoms
# - Sum all the sums that have been calculated in parallel
list
|> Stream.chunk_every(chunk_size)
|> Task.async_stream(&Enum.sum/1)
|> Enum.to_list()
|> Enum.map(&elem(&1, 1))
|> Enum.sum()

For this particular example you should get 5050 as a result.

2 Likes
6

Thanks a lot. This helps really.