Streams for processing a tree of folder and file paths?

I’m not sure about that, however I remember reading something about the possibility of implementing a custom file server, there you should be able to run custom logic.

I’m not aware of anything built-in, but there’s erlang-dirent:

https://github.com/team-telnyx/erlang-dirent

You’d likely want to wrap this in a module that does a couple things:

• translates the input from a binary to a charlist

• uses Stream.resource etc to produce a Stream of matches

• transforms the elements of the stream back into binaries from charlists

This is great thank you!

Forgive me for the naive question.

If I understand this correctly this lib in short before it gives you a full list of the folders/files it can instead give you the folder/file in question as readdir is reading the thing which I assume is how the filesystem builds the full list in the first place? Thus I could chunck that out via Stream.resource

Here’s a standalone demo:

Mix.install([
  {:dirent, git: "https://github.com/team-telnyx/erlang-dirent.git", branch: "master"}
])

target_path = "."

Stream.resource(
  fn ->
    {:ok, dir_ref} =
      target_path
      |> String.to_charlist()
      |> :dirent.opendir()

    dir_ref
  end,
  fn dir_ref ->
    case :dirent.readdir_type(dir_ref) do
      :finished ->
        {:halt, dir_ref}

      {:error, reason} ->
        {[{:error, reason}], dir_ref}

      {name, type} ->
        {[{List.to_string(name), type}], dir_ref}
    end
  end,
  fn _ -> :ok end # not used because :dirent cleans up on GC
)
|> Stream.each(&IO.inspect/1)
|> Stream.run()

Followup thoughts in no particular order:

• This uses :dirent.readdir_type since the first question that code that consumes the stream is likely to ask is “is this a directory?”

• The error handling is somewhat inconsistent; a failure in opendir will crash but a failure when iterating through the results will put {:error, reason} in the output stream. Your application may have different needs.

• the situation with filenames that can’t be represented in UTF8 is complicated so this code completely ignores it. That may not be sufficient depending on your specific filesystem.

I ran into a little issue compiling to otp 26 and drop backed to 25 and it worked fine.
Something to do with IO something or other at build

Any how this is great and works nicely. Thank you.
I can see myself using this along with flow to do the job I’m looking at.