You likely wouldn’t love this idea but I’d suggest you make your own mini-service that handles network-attached directories, maintains a cache, and exposes a network endpoint where the cached lists of files can be served to interested parties (like your Elixir program). It’s also a good idea for it to be written not in Elixir.
Sadly, enumerating big networked directories remains a nasty problem to this day. The original protocols weren’t designed for the usages they see today. Hell, I have one 500GB SD card with about 95 million files. Even doing ls in one of the subdirectories takes like 40 seconds… And I’m not accessing it over network. The filesystems simply cope horribly with slow storage mediums having a ton of files.
If you really have control of the networked directory however, have you considered ditching the files and directories model and putting all files as values in a key-value database (with the “paths” being the keys)?
