Fetching tens of thousands of files regularly

+1 to the recommendation for Oban.

Re: the NIF angle, also consider if ports would be a good fit. You could write a separate Rust program that talks over a port and does the crunching. There’s some overhead - since data is being transferred between operating-system processes - but OTOH if the program on the other end of the port fails it doesn’t affect the BEAM. The NIF approach isn’t free of performance issues related to serialization either.

2 Likes