polypush135

polypush135

Streams for processing a tree of folder and file paths?

I’m looking to learn the File module and was trying to think of a pet project for doing so.

I have a huge dir full of mp3 files that I want to organize.

Given the nature of music Artist → Album → Track.
I figured I would be building a possibly big list of folders and files into memory.
So naturally the first thing that comes to mind is streams.

Is there something like File.ls but for streams?

Or maybe a better question. What kind of limits are there to the number of folders and files that can be made in any one folder on today’s OS systems?

How does File.ls handle large lists in other words?

Marked As Solved

al2o3cr

al2o3cr

I’m not aware of anything built-in, but there’s erlang-dirent:

https://github.com/team-telnyx/erlang-dirent

You’d likely want to wrap this in a module that does a couple things:

  • translates the input from a binary to a charlist

  • uses Stream.resource etc to produce a Stream of matches

  • transforms the elements of the stream back into binaries from charlists

Also Liked

al2o3cr

al2o3cr

Here’s a standalone demo:

Mix.install([
  {:dirent, git: "https://github.com/team-telnyx/erlang-dirent.git", branch: "master"}
])

target_path = "."

Stream.resource(
  fn ->
    {:ok, dir_ref} =
      target_path
      |> String.to_charlist()
      |> :dirent.opendir()

    dir_ref
  end,
  fn dir_ref ->
    case :dirent.readdir_type(dir_ref) do
      :finished ->
        {:halt, dir_ref}

      {:error, reason} ->
        {[{:error, reason}], dir_ref}

      {name, type} ->
        {[{List.to_string(name), type}], dir_ref}
    end
  end,
  fn _ -> :ok end # not used because :dirent cleans up on GC
)
|> Stream.each(&IO.inspect/1)
|> Stream.run()

Followup thoughts in no particular order:

  • This uses :dirent.readdir_type since the first question that code that consumes the stream is likely to ask is “is this a directory?”

  • The error handling is somewhat inconsistent; a failure in opendir will crash but a failure when iterating through the results will put {:error, reason} in the output stream. Your application may have different needs.

  • the situation with filenames that can’t be represented in UTF8 is complicated so this code completely ignores it. That may not be sufficient depending on your specific filesystem.

Last Post!

polypush135

polypush135

I ran into a little issue compiling to otp 26 and drop backed to 25 and it worked fine.
Something to do with IO something or other at build :person_shrugging:

Any how this is great and works nicely. Thank you.
I can see myself using this along with flow to do the job I’m looking at.

Where Next?

Popular in Questions Top

vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New

Other popular topics Top

KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 36654 110
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 49084 226
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
274 42533 114
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

We're in Beta

About us Mission Statement