Can Phoenix web requests be my producer?

Phoenix Forum Questions / Help
,
1

There is an external API with a rate limit and I wanted to use GenStage consumer to ensure that rate limit isn’t hit (let’s call this API consumer).

Questions

  1. I was thinking of having each web request queue a job into AWS SQS and then have the producer-consumer pull the job from SQS then forward the demand to my API consumer. Is there a better way to do this (ex, can I just have the web request produce events for my API consumer directly)?
  2. Is there an Elixir way of queuing jobs that is more idiomatic than using AWS?
  3. Assuming the external API rate limit is per API key and I have multiple keys, what’s the best way to setup my consumers to make sure they each respect their own individual rate limits? would I duplicate the whole pipeline or would I just have a one-to-many relationship (ex, I would have multiple API consumer subscribing to the producer-consumer)?
2

This library does something similar: GitHub - Rob-bie/ElixirPlusReddit: Reddit API wrapper in Elixir

There’s a queue and requests to Reddit are first added to it, and then processed respecting the API limit.

2 Likes
3

Thanks! This looks very similar. I’ll take a look