a-c-sreedhar-reddy

a-c-sreedhar-reddy

Does Enum.map run serially or does it run in parallel?

I have added a task.yield in the callback function of Enum.map, Does Enum.map run serially or in parallel? If it runs serially then this would be a bad idea.

Marked As Solved

al2o3cr

al2o3cr

Enum.map runs for one element at a time - if you have a list of Tasks you’d like to wait for, consider Task.yield_many which will wait for all of them at once.

Also Liked

pejrich

pejrich

Enum functions are for running things serially. It might seem like you’d want to run everything concurrencly in a language that allows for such cheap concurrency, but there’s still a lot of times when serially running is the best choice. Either because logically the steps need to be sequential, or because even though concurrency in Erlang/Elixir is relatively cheap, it’s not free. So, in a contrived example, mapping over the range 1..100 and adding 1 to each element, is about 38x slower when done in parallel. Because we’re doing such a small amount of work in each step, the overhead of concurrency heavily outweighs any benefits of it. In contrast something like mapping over 1..10 and making a request to Google, is about 8x slower sequentially, than in parallel.

So maybe sequential is what you need, in which case Enum.map/2 will be the right tool. If you do want concurrency, something like this is a good starting point. map/2 will start a process for each item in the list, whereas map2/2-4 can be limited, so maybe you want a max of 5 processes started to process 100 things. There’s obviously even more you can do, like supervising the tasks, or more complex processing pipelines. Task is a good place to start. Broadway is a good place to start for the more complex processing.

defmodule PMap do
  def map(enum, fun) do
    Enum.map(enum, &Task.async(fn -> fun.(&1) end))
    |> Task.await_many()
  end
  
  def map2(enum, fun, concurrency \\ 10, timeout \\ :infinity) do
    Task.async_stream(enum, &fun.(&1), max_concurrency: concurrency, timeout: timeout)
    |> Stream.map(fn {:ok, val} -> val end)
    |> Enum.to_list()
  end
end
Nicd

Nicd

Enum.map runs serially, as do all the Enum functions.

hst337

hst337

For parallel processing you can use this

list
|> Task.async_stream(fn x -> x + 1 end)
|> Enum.to_list()

Where Next?

Popular in Questions Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
vac
Hi, I’m quite new in Elixir and I’m trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and I...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1144 53690 245
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New

We're in Beta

About us Mission Statement