We’re using GenStage
in our current code base to push various events to 3rd party tools via HTTP APIs. Like every other API on the web the current 3rd party tool enforces various rate limiting policies on how much we can spam them. We’re using a rate limiter :producer_consumer
stage which is basically a copy of the RateLimiter
module included in the GenStage examples. This rate limiter works fine, however when the 3rd party API returns some error (which honestly is totally fine) everything goes south pretty fast. The problem is our :consumer
stage: It’s pretty dump and just retries sending previously failed requests after a certain (random) timeout. After a while the failed requests pile up and we hit the enforced rate limits because besides sending the failed requests again, we also ask for more events from the :producer
stage and send these new events as well.
I tried to create a separate :producer_consumer
stage which connects between the RateLimiter
and our Consumer
however the problem is that this Retry
module has no clue of the rate limiting happening in the RateLimiter
module. I can ask for 999 instead of the default 1000 events when an error occurs, but how do I communicate to the previous producer-consumer that I want whatever demand it has minus 1?