GenStage in our current code base to push various events to 3rd party tools via HTTP APIs. Like every other API on the web the current 3rd party tool enforces various rate limiting policies on how much we can spam them. We’re using a rate limiter
:producer_consumer stage which is basically a copy of the
RateLimiter module included in the GenStage examples. This rate limiter works fine, however when the 3rd party API returns some error (which honestly is totally fine) everything goes south pretty fast. The problem is our
:consumer stage: It’s pretty dump and just retries sending previously failed requests after a certain (random) timeout. After a while the failed requests pile up and we hit the enforced rate limits because besides sending the failed requests again, we also ask for more events from the
:producer stage and send these new events as well.
I tried to create a separate
:producer_consumer stage which connects between the
RateLimiter and our
Consumer however the problem is that this
Retry module has no clue of the rate limiting happening in the
RateLimiter module. I can ask for 999 instead of the default 1000 events when an error occurs, but how do I communicate to the previous producer-consumer that I want whatever demand it has minus 1?