I need to run an js script 500 times at once and collect the outputs from each execution. Is this something that can easily be done with elixir, at least in comparison to other languages?
I’m also interested (if the above is true) if It is possible to have a separate process track the execution of each of the scripts and update the results on a drab page like in the async tasks example?
Just in case it is important the script that I plan to run is a automated e2e test using selenium.
I have never measured it, but I believe RAM would be a bottleneck for 500 chromes. You may want to run it on the multiple machines, and yes - this is a case for Elixir.
Also, be prepared to have a small number of random bugs occasionally. I am using chromedriver with hound in Drab’s tests, and from time to time it just fails with the different browser errors. Re-running the same test fixes the issue.
There is a service for this, https://www.browserless.io, but they are charging something like 5$ per browser, so probably it would be cheaper to provide your own infrastructure.
Use Task.async_stream/3 and toy with the limit until you get a maximum that fits the hardware you’re on. Once you get that functioning your test will run (which is valuable in itself). Then if that number is unacceptable and runtime is too long, get a bigger box and see what it will give you. Selenium is slow and uses a lot of CPU as well. So its best to get something working as is first before springing to another box so you’ll have something to compare against when you do decide to increase the hardware (not to mention being able to justify or not justify the cost of doing so).
Also, consider starting with number of cores. Don’t try 100 or something crazy like that at first because it will never complete. Start with something with few resource constraints and inefficient and move up to efficient and max resources because once you go past that its just likely to waste your time.