PragTob

PragTob

Blog Post: Careful what data you send or how to tank your performance with Task.async

I ran into an interesting problem recently where simple concurrency on the BEAM via Task.async made my application a lot slower and a lot more memory hungry. This blog post illustrates the issue with a short example, where processing 3 non-trivial actions on a list in parallel is slower than doing it sequentially, and then explains why this happens and what can or can’t be done about it.

Most Liked

dimitarvp

dimitarvp

Come on now. Unless you have a strict one-thread-pinned-per-core runtime then you can’t claim it for any runtime, Golang’s and Rust’s tokio’s included.

Fact is that most parallel runtimes do parallelization on a best-effort basis and they do a damn good job at it. There was a rather hilarious article a while ago reposted on HN how the Linux kernel never used more than 4 (or 8?) cores for a while, some years ago. What can a runtime do if the kernel is lying to it? But that’s a separate topic.

So… 100% guarantee? No, but it’s at least 90%.

lud

lud

No as long as you stay in the same process then you do not care, basically, as the data is not copied. The closure will use the same pointer to the data as the parent scope does.

D4no0

D4no0

Using the term parallel in this context is incorrect, the correct term would be concurrently and this has some very important implications:

  1. You are not guaranteed that the spawned tasks will be running on a separate physical core, as this is decided by the scheduler, there are ways to configure this manually, however you will be breaking the abstraction level of concurrency used in elixir and potentially introduce locks;
  2. As the scheduler switches context between running processes, every process will get a slice of execution time, naturally having more processes on the same thread will make them run slower.

Where Next?

Popular in Blog Posts Top

alexgaribay
I saw this thread so I decided to write about how we do it :smile:
New
luckywatcher
Hey all! I just started a blog focused on web development in Elixir. I wrote my first article and thought I’d post it here for the whole...
New
brainlid
Livebook was created for machine learning in Elixir but Livebook isn’t limited to machine learning. I found it works really well for docu...
New
danschultzer
How to set up the Content Security Policy header with Phoenix LiveView and support inline styles and scripts with CSP nonces.
New
New
rrrene
In this series, we take a look at the different ways to organize, structure and execute a good “flow” in our Elixir programs. The latest...
New
bram209
Hello everyone, I just published my first blog post ever. I am learning Elixir, OTP & Phoenix and wanted to start a blog for a while...
New
hauleth
I wanted to write down some of the main guides I use when writing Elixir tests, so I have summarised them in blogpost. Furthermore, I th...
New
AstonJ
Update: How to use the blogs section You can post in one of the Official Blog Posts threads (like this one), or, via Devtalk and a new t...
New
PragTob
Elixir is a great language, but some behavior can be unintuitive, confusing and in the worst case lead to bugs. So, I took a look at 10 E...
New

Other popular topics Top

albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36352 110
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
klo
Got a question about when to concat vs. prepending items to list then reversing to achieve appending. So i know lists boil down to [1 | ...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement