Fl4m3Ph03n1x

Fl4m3Ph03n1x

How to throttle HTTP requests?

Background

I have a small terminal application that does HTTP requests to a website. However, I am making too many requests per second and I am drowning the receiver.

Solution

The obvious solution is to throttle the rate at which I perform requests.
Since I am using HTTPoison I thought about reading the docs to see if this was a feature, but I didn’t find anything.

The other solution is to find a community library that does this for me. This library would have to not use any external tools (like redis) and throttle out-going requests. After some investigation I didn’t really find anything in the community.

Surely this is something out there and I missed it.

Questions

What libraries do you use when you need to throttle out-going requests?

Marked As Solved

Fl4m3Ph03n1x

Fl4m3Ph03n1x

As a final remark to this thread, I just wanted to say I eventually went with the @cmkarlsson’s suggestion and used the jobs library.

However, I also read @akoutmos’s blog, and I can say that although it is not a reading for beginners, I very much enjoyed it and I can thoroughly recommend it to anyone wanting to learn more about throttlers.

Also Liked

cmkarlsson

cmkarlsson

There is an erlang library for load regulation that I have used in the past.

https://github.com/uwiger/jobs

At the simplest case you can start a job queue which will only allow a specific amount of requests per second. It doesn’t use any external dependencies.

At its simplest:

> jobs.add_queue(:myqueue, [{:standard_rate, 10}])
> jobs.run(:myqueue, fn -> :httpc.request('https://www.forum.elixirforum.com') end)

Runs the jobs at a rate of 10 per second at most

akoutmos

akoutmos

Author of Build a Weather Station with Elixir and Nerves

I actually wrote a blog post about this exact problem and go through the process of implementing token bucket and leaky bucket rate limiters using a simple GenServer and a Task.Supervisor: Easy and Robust Rate Limiting in Elixir-Alex Koutmos | Engineering Blog

I’d imagine that you want to persist your queue in the case that the application goes down. You can add that in pretty easily to the code snippets I walk through by implementing the terminate/2 callback.

Hope that helps!

Where Next?

Popular in Questions Top

hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New

Other popular topics Top

grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 54092 488
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New

We're in Beta

About us Mission Statement