How to create multi thread request

Questions / Help
1

Hello
I need to send a few requests at a time
I know I should use Task.async but I do not know how to use it
Please write me an example if possible

defmodule Test do
  def req(url) do
    case HTTPoison.get(url) do
      {:ok, %{body: body, status_code: 200}} ->
        Jason.decode!(body)
    end
  end
end

list = ["https://sample.com", "https://sample.com", "https://sample.com", "https://sample.com"]

for i <- list do
  Test.req(i)
end
1 Like
2

Why don’t you take a look at the documentation?

1 Like
3

I looked and mentioned above
My main language is not English
Because of that, I’m a little confused about reading the document
I need someone to help and write me an example for the above code

2 Likes
4

One way to achieve making multiple HTTP requests concurrently is to use functions from Task module. This module is part of Elixir standard library, so there’s no need to install anything additional.

Provided this initial code:

defmodule Test do
  def req(url) do
    with {:ok, %{body: body, status_code: 200}} <- HTTPoison.get(url),
         {:ok, decoded_body} <- Jason.decode(body) do
      decoded_body
    end
  end
end

list = ~w(
  https://sample.com
  https://sample.com
  https://sample.com
  https://sample.com
)

…simplest way to spawn multiple concurrent HTTP requests would be to:

for i <- list do
  Task.async(fn -> Test.req(i) end)
end

Or, in case you are interested in collecting responses, then:

tasks =
  for i <- list do
    Task.async(fn -> Test.req(i) end)
  end

results = Task.yield_many(tasks, 5000)

5000 in the last example means the yield_many function will wait up to 5 seconds for all tasks to do their work.

8 Likes
5

Thank you
How can I tell him to do only 10 requests at a time and no more?
For example, if I have a list of 1000, that is, it starts with 1000, how can I tell it to send only 10 to 10 requests?

6

Limiting the number of requests to run at a time is possible by using another function from Task module, one that supports max_concurrency parameter. Let me write an example…

Upd. @gregvaughn beat me to it :sweat_smile::clap:

2 Likes
7 
list
|> Task.async_stream(&req/1, max_concurrency: 10)
|> Enum.to_list()
3 Likes
8

Thank you very much :heart:

1 Like