So what you actually need seems to be some kind of working queue that limits amount of concurrently spawned processes?
Sounds a bit like GenStage or poolboy or… Well, there are plenty, which you need to pick depending on your exact needs.
NobbZ
2
ms68
Yes but the problem is that I don’t know what pool size I need. How should I estimate how many concurrent processes I can spawn based on my machine specifications? Or How should I estimate how many cpu or memory I need for my operation?
0
NobbZ
You do decide this on 2 major factors:
- Trial and Error
- Experience
You can ask for 2., but therefore more about the task and your machine must be known. (Not dev, but actual workhorse)
0
dom
ms68
I should make a queue, I think. I have already set fs.file-max to 500000. But the problem still exists.
0
dom
A quick way to limit how many run in parallel is to use Task.async_stream/5 with its :max_concurrency option. That may be enough if you have only one process kicking off the tasks. Otherwise you could try fancier rate limiting libraries like safetyvalve or jobs.
2
tty
Silly question: did you increase the max open files in the command line shell ? ulimit -n ?
0
outlog
+1 - can you paste in your ulimit? (ulimit -a) - I would think “max user processes” (-u) also comes into play, in this scenario..
0
peerreynders
I suspect that this could be launching 20000 OS processes - so the emfile errors may just be the first manifestation of a much larger problem. Ultimately some sort of concurrency control is needed.
If there is a continuous need to service requests it may be worth looking into GenStage in general and ConsumerSupervisor in particular - where :min_demand, :max_demand can be used to control the level of concurrency.
2
Qqwy
TypeCheck Core Team
What is it what you want to do? Not how you suspect to be able to do it, but what itself is the problem you are attempting to solve? (See the XY problem)
5