How to throttle HTTP requests?

Fl4m3Ph03n1x

Background

I have a small terminal application that does HTTP requests to a website. However, I am making too many requests per second and I am drowning the receiver.

Solution

The obvious solution is to throttle the rate at which I perform requests.
Since I am using HTTPoison I thought about reading the docs to see if this was a feature, but I didn’t find anything.

The other solution is to find a community library that does this for me. This library would have to not use any external tools (like redis) and throttle out-going requests. After some investigation I didn’t really find anything in the community.

Surely this is something out there and I missed it.

Questions

What libraries do you use when you need to throttle out-going requests?

5 comments

#httpoison #throttle

0 2282 5

2020-09-01 09:00:28 UTC

Showing Posts 1 to 5

Show Best Posts
Show All (oldest first)
Show All (newest first)

lalo2302

Check out HTTPoison pools.
You can read about GenStage. This basically covers your case although it might be an overkill, it depends on your use case
You can easily play around with a GenServer, having just a counter of requests being made. This can lead to some mistakes, but it might suit your needs.

There might be other libraries around which can handle this, but you can do this hands on and should not be very difficult.

Post #1

Fl4m3Ph03n1x

Those are fair solutions, but if I wanted to do this myself I would have used an ETS table to queue up the requests as and then make batch requests to the target website.
I am just trying to avoid the workload.

Post #2

cmkarlsson

There is an erlang library for load regulation that I have used in the past.

https://github.com/uwiger/jobs

At the simplest case you can start a job queue which will only allow a specific amount of requests per second. It doesn’t use any external dependencies.

At its simplest:

> jobs.add_queue(:myqueue, [{:standard_rate, 10}])
> jobs.run(:myqueue, fn -> :httpc.request('https://www.forum.elixirforum.com') end)

Runs the jobs at a rate of 10 per second at most

Post #3

akoutmos

Author of Build a Weather Station with Elixir and Nerves

I actually wrote a blog post about this exact problem and go through the process of implementing token bucket and leaky bucket rate limiters using a simple GenServer and a Task.Supervisor: Easy and Robust Rate Limiting in Elixir-Alex Koutmos | Engineering Blog

I’d imagine that you want to persist your queue in the case that the application goes down. You can add that in pretty easily to the code snippets I walk through by implementing the terminate/2 callback.

Hope that helps!

Post #4

Fl4m3Ph03n1x

As a final remark to this thread, I just wanted to say I eventually went with the @cmkarlsson’s suggestion and used the jobs library.

However, I also read @akoutmos’s blog, and I can say that although it is not a reading for beginners, I very much enjoyed it and I can thoroughly recommend it to anyone wanting to learn more about throttlers.

Post #5

— All posts loaded —