What is the indication of async-threads when starting iex?

What is the indication of async-threads when starting iex?
I couldn’t found anything on the web. I have 8 cores on my machine
and when I launch iex, it prints out async-threads:1
Is there a way to manually increase it?

That’s a setting related to your Erlang installation, can you please show the full headline and tell us what operating system you are using as well as how you installed Erlang and elixir?

Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Interactive Elixir (1.6.6) - press Ctrl+C to exit (type h() ENTER for help)

I am using ubuntu 18.04 and I installed erlang and elixir following the offcial docs. I have found a way to increase it by passing --erl “+A number_of_threads” to iex. But I’m still confused about the limitation. How many async_threads is normal for an 8 core cpu?

You have 8 schedulers online, so you can use the full power of your CPU.

async-threads is mostly for IO related stuff as I understood it.

You can read more on SO:

1 Like

Since OTP 21 none of the built-in things use async threads (they migrated to dirty schedulers and dirty NIFs), so it doesn’t make much sense to spawn those threads. There’s still one of them in case you have some custom linked-in port drivers using them.

On previous versions the file ports were using async threads.

6 Likes

That’s what I’m trying to do. I want to to run about 20000 System.cmd concurrently. Is there any use in increasing async_threads? If yes, how can I calculate what is the appropriate number based on my cpu cores?

System.cmd spawns external processes. They don’t run inside the Erlang VM, so they don’t use async threads.

3 Likes

Do you have any suggestion?

What I am trying to do with System.cmd is running an I/O bound operation (snmp request). Do you think if I use an snmp library written in erlang/elixir, I will achieve better concurrency?

Possibly, but what problem are you hitting exactly?

Concurrency. I will get lots emfile errors after spawning 20000 System.cmd functions.

So what you actually need seems to be some kind of working queue that limits amount of concurrently spawned processes?

Sounds a bit like GenStage or poolboy or… Well, there are plenty, which you need to pick depending on your exact needs.

2 Likes

Yes but the problem is that I don’t know what pool size I need. How should I estimate how many concurrent processes I can spawn based on my machine specifications? Or How should I estimate how many cpu or memory I need for my operation?

You do decide this on 2 major factors:

  1. Trial and Error
  2. Experience

You can ask for 2., but therefore more about the task and your machine must be known. (Not dev, but actual workhorse)

What did you try to fix the emfile errors?

I should make a queue, I think. I have already set fs.file-max to 500000. But the problem still exists.

A quick way to limit how many run in parallel is to use Task.async_stream/5 with its :max_concurrency option. That may be enough if you have only one process kicking off the tasks. Otherwise you could try fancier rate limiting libraries like safetyvalve or jobs.

2 Likes

Silly question: did you increase the max open files in the command line shell ? ulimit -n ?

+1 - can you paste in your ulimit? (ulimit -a) - I would think “max user processes” (-u) also comes into play, in this scenario…

I suspect that this could be launching 20000 OS processes - so the emfile errors may just be the first manifestation of a much larger problem. Ultimately some sort of concurrency control is needed.

If there is a continuous need to service requests it may be worth looking into GenStage in general and ConsumerSupervisor in particular - where :min_demand, :max_demand can be used to control the level of concurrency.

2 Likes