Fl4m3Ph03n1x

Fl4m3Ph03n1x

How to throttle HTTP requests?

Background

I have a small terminal application that does HTTP requests to a website. However, I am making too many requests per second and I am drowning the receiver.

Solution

The obvious solution is to throttle the rate at which I perform requests.
Since I am using HTTPoison I thought about reading the docs to see if this was a feature, but I didn’t find anything.

The other solution is to find a community library that does this for me. This library would have to not use any external tools (like redis) and throttle out-going requests. After some investigation I didn’t really find anything in the community.

Surely this is something out there and I missed it.

Questions

What libraries do you use when you need to throttle out-going requests?

Marked As Solved

Fl4m3Ph03n1x

Fl4m3Ph03n1x

As a final remark to this thread, I just wanted to say I eventually went with the @cmkarlsson’s suggestion and used the jobs library.

However, I also read @akoutmos’s blog, and I can say that although it is not a reading for beginners, I very much enjoyed it and I can thoroughly recommend it to anyone wanting to learn more about throttlers.

Also Liked

cmkarlsson

cmkarlsson

There is an erlang library for load regulation that I have used in the past.

https://github.com/uwiger/jobs

At the simplest case you can start a job queue which will only allow a specific amount of requests per second. It doesn’t use any external dependencies.

At its simplest:

> jobs.add_queue(:myqueue, [{:standard_rate, 10}])
> jobs.run(:myqueue, fn -> :httpc.request('https://www.forum.elixirforum.com') end)

Runs the jobs at a rate of 10 per second at most

akoutmos

akoutmos

Author of Build a Weather Station with Elixir and Nerves

I actually wrote a blog post about this exact problem and go through the process of implementing token bucket and leaky bucket rate limiters using a simple GenServer and a Task.Supervisor: Easy and Robust Rate Limiting in Elixir-Alex Koutmos | Engineering Blog

I’d imagine that you want to persist your queue in the case that the application goes down. You can add that in pretty easily to the code snippets I walk through by implementing the terminate/2 callback.

Hope that helps!

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New

Other popular topics Top

baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New

We're in Beta

About us Mission Statement