fchabouis
Good solution to launch asynchronous tasks with rate limiting
Hello,
This is my first question here, please let me know if it can be improved.
Every night, I would like my server to run a validation process on a number of urls. It computes some hash, download some data, validate it, etc.
It looks like tasks are best suited for this. I’ve also read that supervised tasks are better than just creating unsupervised tasks, for the clean-up, in case the main process is terminated. Can you confirm that I’m taking a good direction so far ?
Also, my server is running on a small instance, I’m worried that if I give it a big list of tasks, it will just crash because of the small resources I have available. I was hoping that the Task.Supervisor would have an option about the maximum number of concurrent tasks running, but I didn’t see any. Any information about this ?
I saw async_stream had a concurrency limit option, but didn’t understand if it could be helpful for my situation. Is it ? And is it compatible with a Task.Supervisor ?
Probably many newbie questions here, thanks for your time !
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #elixirconf-us
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 2 of 2 Posts!
hauleth
Do not worry. BEAM uses preemptive scheduler (at least from the viewpoint of the user) so it should be ok to spawn new process per each URL (depending on the scale) and just let VM do its magic. Erlang processes are quite cheap, so this should not be a problem if there is list of 100s of URLs, I would start thinking about any throttling only if there would be over 1k of such. In that case you can use
GenStagewhich allows for rate limiting.gregvaughn
I use
Task.Supervisor.async_stream_nolink/6in several places for this sort of thing. The tasks are supervised by the supervisor in the first param, but they are not linked to the caller. In the caller, I thenEnum.reduceover the resulting stream and collect a little “report” of number of successes/errors/etc and log the result.I typically specify the concurrency and unordered settings in the options. This concurrency throttling mostly helps me to not overwhelm some external API that the tasks call. It’s not because I’m worried about the BEAM handling a large number of tasks.