How to do massive concurrent http requests?

Hello, can anyone point me to an article/post on how to do concurrent requests in Elixir? Here is an article where the author uses python:

I am also interested in how you would limit the processes to a certain number as to not overwhelm the server or hit a rate limit.

Thanks!

What about Task.async_stream: Task — Elixir v1.12.3

1 Like

I will check it out, I am just getting into Elixir. In fact, Elixir in Action is in the mail as we speak. I guess my question arises because Python has to do it this way, but it seams Elixir has more option available for concurrency and I want to get an idea of the best approach for this common task.

async_stream is the most standard way to to concurrent tasks with a limit on the concurrency, as far as I know. There are a few version of them

  • Task.async_stream
  • Task.Supervisor.async_stream
  • Task.Supervisor.async_stream_no_link

But to say which one is “best” depends on your situation. You would need to learn more about Elixir to know if you want supervision or linking.

Thank you! I just want to play with something until the book gets here. I appreciate the nudge in the right direction.