Path module timing out , listing network file sharewith many files


gets a
** (exit) exited in:, {:list_dir, ‘//’})
** (EXIT) time out
(stdlib) gen_server.erl:215:
(elixir) lib/path.ex:590: Path.Wildcard.list_dir/1
(stdlib) filelib.erl:285: :filelib.do_wildcard_3/4
(stdlib) filelib.erl:278: :filelib.do_wildcard_2/4
(stdlib) filelib.erl:260: :filelib.do_wildcard/3
(stdlib) filelib.erl:62: :filelib.wildcard/2
(elixir) lib/path.ex:657: Path.wildcard/2

how do i override the time out ?


You can’t. It might help though to reduce the number of files in the folder or indexing them by other means, or perhaps even providing faster storage.

Thanks for the reply
Will have to see what can be done

You likely wouldn’t love this idea but I’d suggest you make your own mini-service that handles network-attached directories, maintains a cache, and exposes a network endpoint where the cached lists of files can be served to interested parties (like your Elixir program). It’s also a good idea for it to be written not in Elixir.

Sadly, enumerating big networked directories remains a nasty problem to this day. The original protocols weren’t designed for the usages they see today. Hell, I have one 500GB SD card with about 95 million files. Even doing ls in one of the subdirectories takes like 40 seconds… And I’m not accessing it over network. The filesystems simply cope horribly with slow storage mediums having a ton of files.

If you really have control of the networked directory however, have you considered ditching the files and directories model and putting all files as values in a key-value database (with the “paths” being the keys)?