I have I think a common problem similar to How to throttle HTTP requests? - #4 by cmkarlsson but I wanted to ask for some advices to help make sure I understand the best approach.
I have an app that makes web requests to many different web sites and APIs. I have made a behaviour with a get callback – go get a thing! So there are many getters that must get and parse things – sometimes a JSON API, sometimes scrape an HTML page.
There are some business rules around these getters: 1. some of them must throttle requests (because of API usage or server limits) 2. there are special rules for when to retry requests (for example, to retry when receive 500 request and how many times and how long to wait). So this is more complicated than just “retry 5 times” or something. 3. no duplicate requests
To have rules about the number of requests and the re-try, I must have state, so I thought I could use a GenServer to handle looking up information about a URL. If I have multiple modules casting to this GenServer, then the handle_cast can do the HTTP gets using my behaviour implementations. But I am worried because handle_cast still deals with each message one at a time. Is there a better solution where this “request manager” can handle x parallel HTTP requests? I feel like this maybe is the job for a supervisor
I thought the logics around re-try requests would be easier with separate processes for each request. Is this what Task.Supervisor is for? On success, I only need to send a PubSub message, so I think handle_cast is ok for this.
Sorry I cannot explain this clearly… I feel like there are many bad ways to do this but I don’t even understand what is good or bad. Thank you for any guidance!
