System.cmd output in a stream

Questions / Help
1

Is there a fairly direct way to get from System.cmd a stream which when enumerated will return the command output line by line?

:into IO.stream()

Just echos output to stdout as it comes in. It seems that I can get what I need by either creating a process to receive output and using receive there, or dropping down to Erlang and using a port directly. But I’m wondering if there is some higher-level support for this which I am missing?

2

:into parameter accepts Collectable.t(), so you can use an empty list:

> System.shell("echo hello; sleep 1; echo world; sleep 1; echo end;", into: [])
{["hello\n", "world\n", "end\n"], 0}

or create a custom struct and implement Collectable protocol for it.

2 Likes
3 
> System.shell("echo 1; echo 2; echo 3", into: [])
{["1\n2\n3\n"], 0}

Sleeping between every line of output doesn’t cut it :wink:

(Especially not for System.cmd where I specifically want a stream of lines from 1 OS process.)

4

As far as I know, there’s no more higher-level abstractions in Elixir itself.

You have to:

  • use Port directly
  • or, some external packages, like porcelain

Personally, I prefer using Port directly, you can get more examples at:

I’m not sure about the context of your requirements. But personally, I always lean towards small and sufficient implementations rather than large and comprehensive ones. From that perspective, a simple Port wrapper is sufficient.

2 Likes
5

This is the default behaviour of port.
You have to explicitly pass {line, L} option, but this requires more work to be done, as L is a maximum length of line, so you have to deal with :noeol messages.
It is far easier to apply String.split/3 to split the string by \n

6

Yep. I’ve gone through the source for System.cmd and the documentation for port, and I see how to do it that way. I’m thinking some support in System.cmd for a couple more port options might be useful. For the moment I’ll deal with the “one chunk” return, and circle back to optimizing later–and see if I think it would be useful to make a PR against System.cmd.

7

I note that L can be 0. Does that imply no limit? Docs don’t say what it means; I will test later when I have time. If it does, that’s ideal for this use.

Gulping millions of lines into a string and then splitting works, but the point of streaming by line is to avoid unnecessarily building a humongous stream.

8

Gulping millions of lines into a string and then splitting works, but the point of streaming by line is to avoid unnecessarily building a humongous stream.

This is actually how the ports are implemented (and hence System.cmd). Internally, port just reads everything that is available on the stdout buffer and send it as a message to the calling process, it does not wait for the caller to consume. Also, the config {line, L} how the output is delivered to calling process not how it is read into VM. So if your external program producing a lot of output in short amount of time then it still can get accumulated in your beam vm memory irrespective of the line limit config.

You can test it out in iex with observer.

iex(1)> :observer.start
:ok
iex(2)> Port.open({:spawn_executable, "/bin/cat"}, [{:args, ["/dev/random"]}, {:line, 100}, :binary, :use_stdio])
#Port<0.5>

For most of the use case, it is not a big issue. If it is, then there are third party libraries which solves in different ways.