Hey Guys,
So in my day job I need to retrieve allot of data through an external API that is rate limited. I do the retrieving/consuming with the help of PHP and the Laravel framework. I feel like I could do this job way better using Elixir (with or without Phoenix).
The way it works now is by keeping a table called resources with in every row a specific resource (endpoint). Every row contains the query parameters I need to remember, like the lastId (the id of the last retrieved resource). I also have a table containing all my keys. Usually I only have 1, but I could have multiple as well. The number of calls I can make per key is usually a 1000 calls per 5 min otherwise it’s 300 per 5 min.
The way of knowing my current consumed call’s and when its about to reset can be found out by looking at the X-RateLimit-* headers thats added to every response from the API. There is also a dedicated API call to retrieve those in the response body.
X-RateLimit-Limit →1000
X-RateLimit-Remaining →999
X-RateLimit-Reset →40 (in seconds)
So my question is what techniques, modules, techniques and/or tips do you guys have for me around using Elixir for consuming the API. This is going to be a long running operation cause I needed to retrieve allot of data (for example I got a resource with 2-3 million records. I get 250 per call.). Once I retrieved every resource up to a point, I would only need it to keep adding the new ones made (polling every 1-5 minutes?).
Things I have done so far:
Made a prototype mix module for consuming the API (github Ilyes512/lightspeedex).
BTW there is more to the story and I kept allot of it from the above, but if you need more info just let me know :).
