While working on the ICFP Contest this weekend, I struggled to find a good way to periodically read content not delimited by newlines. Let me show some examples of what I mean.

Here’s a Ruby script that produces messages:

$stdout.sync = true
loop do
  message = rand(1_000_000)
  $stdout.write "<#{message}>"
  sleep rand(3) + 1
end

I can think of multiple ways, using Ruby, to read these messages as they come in. For example, I can use non-blocking I/O:

loop do
  begin
    raw = $stdin.readpartial(1_024)
    puts raw[/\d+/]
  rescue EOFError
    sleep 0.1
  end
end

Or I can read what’s available:

require "io/wait"

loop do
  $stdin.wait_readable
  raw = $stdin.read($stdin.nread)
  puts raw[/\d+/]
end

There are other options too, like using IO::select(). Here’s how the above examples work in practice:

$ ruby producer.rb | ruby read_nonblocking.rb 
319187
122221
30420
…
$ ruby producer.rb | ruby read_ready.rb 
640243
971582
366808
…

I haven’t found a good way to do similar work with Elixir. The best I’ve come up with for the same input is to read character by character:

defmodule MessageReader do
  def read_message(device, buffer \\ "") do
    new_buffer = buffer <> IO.read(device, 1)
    if String.first(new_buffer) == "<" and String.last(new_buffer) == ">" do
      String.slice(new_buffer, 1..-2)
    else
      read_message(device, new_buffer)
    end
  end

  def read_messages(device, handler) do
    read_message(device)
    |> handler.()
    read_messages(device, handler)
  end
end

MessageReader.read_messages(:stdio, &IO.puts/1)

This does work:

$ ruby producer.rb | elixir read_chars.exs  
963609
378034
387827
…

However, that would be pretty inefficient with long messages and I can’t find a way to read ahead. Am I missing a useful trick?

Thanks in advance!