Long time lurker here

Some background: I’ve been learning and playing with Elixir for about a year now. I did a few simpler projects with Phoenix, but now I wish to try something a bit harder.

As the title suggests one of the parts of the system I’m trying to build is going to fetch thousands of files regularly, perform some form of processing and then save any relevant information.

I think I have most of the parts figured out. There are plenty of good resources of doing APIs + DB and making that handle lots of requests from the outside.

What is the best way to handle this tho when the application itself generates that amount of work? For doing the processing I’m looking at Rust NIF, which looks promising & I’m curious to play around with it. My question is more around how to handle the logic around a large (and let’s imagine increasing) volume of work. I’ve looked at GenStage, but I’m not sure I understand enough to imagine how it would work. What would be the tradeoffs in Elixir between getting the work done fast making sure the app doesn’t crash itself.

Another thing I’m trying to wrap my head around is: How to write this without limiting the application in the way that it’s written. If a worker pool is used that might work, but what if the app is hosted on GCP or AWS and the machine can be scaled vertically, the app wouldn’t scale without changing the code?

How can I exploit the concurrency here optimally? How does one determine the boundaries in such cases and then make sure they are respected?

I’m not sure I’m explaining well enough here, that’s probably due to my level of understanding =)