There is an external API with a rate limit and I wanted to use GenStage consumer to ensure that rate limit isn’t hit (let’s call this API consumer).
- I was thinking of having each web request queue a job into AWS SQS and then have the producer-consumer pull the job from SQS then forward the demand to my API consumer. Is there a better way to do this (ex, can I just have the web request produce events for my API consumer directly)?
- Is there an Elixir way of queuing jobs that is more idiomatic than using AWS?
- Assuming the external API rate limit is per API key and I have multiple keys, what’s the best way to setup my consumers to make sure they each respect their own individual rate limits? would I duplicate the whole pipeline or would I just have a one-to-many relationship (ex, I would have multiple API consumer subscribing to the producer-consumer)?