managua1902

managua1902

How to properly limit my own rate requests to external API?

I use Oban to make external API requests every N minutes. It’s turned out that the frequnecy I use it with is too dense such that it’ll cause an error “too many requests” from API, as there’s a limit there.

How would I properly adjust my requests? I could a) incease N b) insert a few Process.sleep(10_000) inside the code that makes the requests.

An issue with (a) is that as my DB grows, it’ll have to be increasing N too, doing it by trial and error.

An issue with (b) is that the next iteration of Oban.perform(...) may coincide as the current one is getting executed, if Process.sleep(M) is too large. Otherwise, it may cause an API limit exceeded error.

How to do this properly? And in a simple manner too.

Most Liked

sorenone

sorenone

Oban Core Team

As we don’t actively proselytize, this is a marvelous compliment. We’ll take it!

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Can you elaborate on why you think this?

Oban Pro has rate limiting Smart Engine — Oban v2.11.0. That however operates at the job level not the API request level. However if these are pretty 1:1 that might work fine.

mbklein

mbklein

I don’t know much about how Oban works under the hood, but you could see if there’s a way to integrate Hammer into the execution loop. It’s a rate-limiter for Elixir that acts as a kind of traffic signaling device.

Last Post!

brady131313

brady131313

I ran into a similar issue needing to rate limit my own requests. What I did was use ExHammer to check if I’m over or under the limit and from there can snooze the Oban job for however long until the next bucket is available to make more requests.

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 130286 1222
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
AngeloChecked
What learn first? Rust or Elixir Hi Elixir community! I’m here because i want learn a new language. I’m a junior developer and mainly i ...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New

We're in Beta

About us Mission Statement