a-c-sreedhar-reddy

a-c-sreedhar-reddy

Does Enum.map run serially or does it run in parallel?

I have added a task.yield in the callback function of Enum.map, Does Enum.map run serially or in parallel? If it runs serially then this would be a bad idea.

Marked As Solved

al2o3cr

al2o3cr

Enum.map runs for one element at a time - if you have a list of Tasks you’d like to wait for, consider Task.yield_many which will wait for all of them at once.

Also Liked

pejrich

pejrich

Enum functions are for running things serially. It might seem like you’d want to run everything concurrencly in a language that allows for such cheap concurrency, but there’s still a lot of times when serially running is the best choice. Either because logically the steps need to be sequential, or because even though concurrency in Erlang/Elixir is relatively cheap, it’s not free. So, in a contrived example, mapping over the range 1..100 and adding 1 to each element, is about 38x slower when done in parallel. Because we’re doing such a small amount of work in each step, the overhead of concurrency heavily outweighs any benefits of it. In contrast something like mapping over 1..10 and making a request to Google, is about 8x slower sequentially, than in parallel.

So maybe sequential is what you need, in which case Enum.map/2 will be the right tool. If you do want concurrency, something like this is a good starting point. map/2 will start a process for each item in the list, whereas map2/2-4 can be limited, so maybe you want a max of 5 processes started to process 100 things. There’s obviously even more you can do, like supervising the tasks, or more complex processing pipelines. Task is a good place to start. Broadway is a good place to start for the more complex processing.

defmodule PMap do
  def map(enum, fun) do
    Enum.map(enum, &Task.async(fn -> fun.(&1) end))
    |> Task.await_many()
  end
  
  def map2(enum, fun, concurrency \\ 10, timeout \\ :infinity) do
    Task.async_stream(enum, &fun.(&1), max_concurrency: concurrency, timeout: timeout)
    |> Stream.map(fn {:ok, val} -> val end)
    |> Enum.to_list()
  end
end
Nicd

Nicd

Enum.map runs serially, as do all the Enum functions.

hst337

hst337

For parallel processing you can use this

list
|> Task.async_stream(fn x -> x + 1 end)
|> Enum.to_list()

Where Next?

Popular in Questions Top

Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
JulienCorb
I am trying to implement my new.html.eex file to create new posts on my website. new.html.eex: <h1>Create Post</h1> <%= ...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New

Other popular topics Top

sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36128 110
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
PeterCarter
There are pre-rolled solutions for other frameworks that do work. However, Phoenix does not seem to have these. Have people had good expe...
New

We're in Beta

About us Mission Statement