polypush135
Streams for processing a tree of folder and file paths?
I’m looking to learn the File module and was trying to think of a pet project for doing so.
I have a huge dir full of mp3 files that I want to organize.
Given the nature of music Artist → Album → Track.
I figured I would be building a possibly big list of folders and files into memory.
So naturally the first thing that comes to mind is streams.
Is there something like File.ls but for streams?
Or maybe a better question. What kind of limits are there to the number of folders and files that can be made in any one folder on today’s OS systems?
How does File.ls handle large lists in other words?
Marked As Solved
al2o3cr
I’m not aware of anything built-in, but there’s erlang-dirent:
https://github.com/team-telnyx/erlang-dirent
You’d likely want to wrap this in a module that does a couple things:
-
translates the input from a binary to a charlist
-
uses
Stream.resourceetc to produce aStreamof matches -
transforms the elements of the stream back into binaries from charlists
Also Liked
al2o3cr
Here’s a standalone demo:
Mix.install([
{:dirent, git: "https://github.com/team-telnyx/erlang-dirent.git", branch: "master"}
])
target_path = "."
Stream.resource(
fn ->
{:ok, dir_ref} =
target_path
|> String.to_charlist()
|> :dirent.opendir()
dir_ref
end,
fn dir_ref ->
case :dirent.readdir_type(dir_ref) do
:finished ->
{:halt, dir_ref}
{:error, reason} ->
{[{:error, reason}], dir_ref}
{name, type} ->
{[{List.to_string(name), type}], dir_ref}
end
end,
fn _ -> :ok end # not used because :dirent cleans up on GC
)
|> Stream.each(&IO.inspect/1)
|> Stream.run()
Followup thoughts in no particular order:
-
This uses
:dirent.readdir_typesince the first question that code that consumes the stream is likely to ask is “is this a directory?” -
The error handling is somewhat inconsistent; a failure in
opendirwill crash but a failure when iterating through the results will put{:error, reason}in the output stream. Your application may have different needs. -
the situation with filenames that can’t be represented in UTF8 is complicated so this code completely ignores it. That may not be sufficient depending on your specific filesystem.
Last Post!
polypush135
I ran into a little issue compiling to otp 26 and drop backed to 25 and it worked fine.
Something to do with IO something or other at build ![]()
Any how this is great and works nicely. Thank you.
I can see myself using this along with flow to do the job I’m looking at.
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security









