Using Process.sleep for rate limiting external requests?

[_jonas]

Well I don’t know how they implement their rate limiting, could be a rolling time window, or could be looking at the timestamps of the 240 previous requests or something, not sure.

@dimitarvp

Say you do 60 requests between 13:20:00 and 13:20:15, now you have to wait 45 seconds until 13:21:00 comes around?

Not sure I understand, the rate limit is 240 per minute, so if I do 60 requests between 13:20:00 and 13:20:15, I have another 45 seconds to do 180 requests before I hit the rate limit and am required to wait. But wait time is a full minute yeah.

I could already throttle my requests on my side to never reach the rate limit which would probably be more time efficient if that’s what you’re getting at, but I’m happy with the low level of complexity I have at the moment, because as mentioned I’m not expecting to have high sustained load that I need to process as quickly as possible.

[dimitarvp]

Ah sorry I gave the wrong number. I meant 240 not 60.

And yep you should probably clear it up with them because waiting until the next period is one policy, and continuously counting how many requests you have for the last X seconds / minutes is another. You should have it said what policy do they use.

[jswanner]

@_jonas

No I want to absolutely never drop requests

In that case, you might want to use Oban or the like. You’ll lose your message queue on restarts (like deployments).

[schneebyte]

Might be easier to add a last_sync timestamp to the database and just query for all rows where last_sync is before last_update.

[dimitarvp]

Could be interesting to share what solution will you eventually arrive at btw.