What's good way to have limits for calling external APIs on a **monthly** or daily level?

what would be easy to set up is a db chart of location, current temperature, date & time, clever remarks id
Then when the user lands on the page, a background query of that location’s date&time compare, then if it is within a certain amount of time since the last query of the temperature it displays the data you want on that chart line, if not, query the parse rss feed and update the temperature and the stored comments for the current weather, which would work well on a separate db table (like a remarks table with a unique column id) from the weather db table, a compare db table can be generated per location over time, and of course the remarks from another db chart.

Parsing data from a free xml/rss feed might be a better way of doing above instead of a web API.

I think you’re overthinking this. Write code that does what you described. I’d start with a database table that stores some token/label that represents a “bucket” of limits and a timestamp. Every time an API call is made you write an entry to that table. Before making the call you query for the limits – group by the various time intervals with a count of each and a having clause to enforce your desired constraints. It won’t be a trivial query (possibly window function or lateral join) but you’ll learn good things figuring that out. In the end it’ll be a single query.

I would highly encourage you to write this as its own package. This will be a great way to keep your concerns separated. I also think there are probably many others folks that could use a utility exactly like this!

I am with @gregvaughn here, you are over-thinking it a bit.

Having your own GenServer that’s part of the app’s supervision tree is what I would do. You can either:

1. Persist every single usage of the API in the database and then aggregate as @gregvaughn advised, so you know how much you have remaining and what waits must be enforced before the next call;
2. Or persist a monthly digest which is just a single SQL record e.g. month="2022-10" and api_calls=174, and then just update that from your supervised GenServer on each API call. Write a wrapper to the API that (a) sends a message to the GenServer and the GenServer’s job is (1) wait until a quota is met, (2) increase the counter in the DB, and (c) actually call the API.

Example 1: if the budget you allocated says you have only 2 requests remaining during the current minute while 10 in total are allowed, and you have 10 seconds remaining, well, enforce a :timer.sleep(5_000) (10 seconds divided by 2 requests) before calling the API.

Example 2: if the minute is ending in 5 seconds but you still haven’t used even 1 out of the 10 allowed requests for this minute then you should do :timer.sleep(500) (5 seconds divided by 10 requests e.g. 500ms).

That’s a super naive approach however. It can be improved and made much more flexible.

It does not look very hard. Like many programming problems the true concern is (a) clearing up the entire problem in your head and (b) figure out what data shape will serve you best. Actually coding it is rarely the challenge.

Give it a go and show us what you tried if it’s not quite happening still. I am positive the people will help you finalize it.

Its not my project. But I would think getting the raw data from a free xml/rss feed would probably be better than someone else’s web API that you don’t have control over it and might change over time.

Seems I pressed Reply to the wrong person.

Sorry!

@artem My above long comment is directed at you.

Agreed. I focused on posting thoughts related to the API route is all.

If you can get the data without having to constantly ping an external service then that would be ideal.

The only thing I want to add is to separate the concern of avoiding abuse from the concern of budget control. Use the database approach similar to what @gregvaughn have suggested to address your budget concern, and use a GenServer approach similar to what @dimitarvp have suggested to address the abuse avoidance concern.

Thank you, guys. I was also considering just writing a log of requests to the database and using SQL queries to check whether bucket is full or not. My main concern (that I should have stated more clearly in the beginning) was:

• Am I reinventing the wheel? There could already be some known library to do just what I want or something very similar. Like if that Hammer library had a Postgres backend functional out of the box, it could be cheaper and more proper to just tune it to my case.

Now after this discussion it seems like my problem either isn’t common enough to have some known solutions or so simple that nobody bothered to make a library for it.

I guess most of hobbyists are using either totally free tiers or ones that have a clear spending limit. One I am likely to use is “metered”, so I could spend too much by accident. Probably it is not very common.

You very likely are. However I’d still go for writing this on your own, because:

1. You will have 100% of it inside your own codebase and it can be tweaked, changed or outright removed very quickly.
2. Figuring out how to configure the already-coded library might take longer and/or be more confusing than writing it yourself.
3. Yes, it’s quite a simple thing and it’s very possible that many people didn’t consider it worthwhile to create and open-source it. I don’t claim it as a fact because I haven’t checked Hammer, I’m merely saying that this is the general culture of the Elixir community.

Again, don’t be shy sharing a partially working code. I believe people will be curious and will help you finish it if it proves more challenging than we think right now.