Streaming a log text file

Phoenix Forum Questions / Help
1

This may be more of an elixir question, but I am trying to stream a log file to web page through Phoenix.

I have a page with a JS Phoenix Socket subscribed to a topic on a channel.

My question is how should I go about publishing the content from the local log file as it comes in.

2

Here is a library that lets you receive messages whenever a file changes. You could use a genserver (like in the example) and respond to the notification messages by reading the new lines and sending them down the channel. Alternatively you could poll the file on some interval and check if the file size has changed.

Reading the new lines from the file might be slightly trickier. You don’t want to have to reread the entire file every time. You should be able to open a file pointer with File.open that you can keep around and then read from it using IO.binread, which keeps track of the position. IO.binread lets you read lines and will return an :eof when the device contents have been consumed.

In my mind it would look like this:

  1. Have a genserver that sends itself a message at some polling interval (say 15 seconds?) (Process.send_after(self(), :read_log, 15 * 1000)).

  2. Respond to the message by reading from the file pointer until :eof if encountered, buffering the results

  3. Send the new lines (if any) down the phoenix channel

This is one way I could picture it working. There might be easier ways to accomplish this, or it might not even work at all. I’m curious to see how it would work so I might give a shot myself.

1 Like
3

I threw an example together (also below) of how this might work. The LogWriter is only for testing purposes, it just periodically writes random lines to the log file in order to mimc a real log. LogReader will periodically consume all new lines in a log file and do something with them. I have them writing to stdout in the example, but this could be sending them a channel or another service.

I’m not sure what sort of performance implications (if any) are involved with this. This is more of a proof of concept. You might be better off using the file_system library so that you get notified directly of new lines instead of having to poll.

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Log Reader - polls log file and sends new lines to channel
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
defmodule LogReader do
  use GenServer

  @log_file "priv/some_log.log"
  @poll_interval 5 * 1000 # 5 seconds

  def run_test() do
    {:ok, _} = LogWriter.start_link()
    {:ok, _} = LogReader.start_link()
  end

  def start_link(_ \\ []) do
    GenServer.start_link(__MODULE__, :ok, name: LogReader)
  end

  def init(:ok) do
    # open the log file and set the pointer to the end so that we only grab
    # new log messages
    {:ok, fp} = File.open(@log_file, [:read])
    :file.position(fp, :eof)
    poll()
    {:ok, fp}
  end

  def handle_info(:read_log_lines, fp) do
    # consume any new log lines and pass them off to the channel
    fp |> read_til_eof |> send_to_channel
    poll()
    {:noreply, fp}
  end

  def read_til_eof(fp),
    do: read_til_eof(IO.binread(fp, :line), fp, [])
  def read_til_eof(:eof, _fp, buffer), do: buffer
  def read_til_eof(line, fp, buffer),
    do: read_til_eof(IO.binread(fp, :line), fp, buffer ++ [line])

  # this could be handing off the new lines to some service or sending directly
  # to the channel or whatever
  def send_to_channel([]), do: :ok
  def send_to_channel(lines),
    do: for line <- lines, do: IO.puts line

  defp poll(),
    do: Process.send_after(self(), :read_log_lines, @poll_interval)

end

#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Log Writer - simulates log writes
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
defmodule LogWriter do
  use GenServer

  @log_file "priv/some_log.log"

  def start_link(_ \\ []) do
    GenServer.start_link(__MODULE__, :ok, name: LogWriter)
  end

  def init(:ok) do
    {:ok, fp} = File.open(@log_file, [:append])
    poll()
    {:ok, fp}
  end

  def handle_info(:write_log, fp) do
    for _ <- 0..:rand.uniform(5), do: IO.puts fp, make_log_message()
    poll()
    {:noreply, fp}
  end

  def make_log_message() do
    time = :os.system_time(:milli_seconds) |> to_string()
    body = :crypto.strong_rand_bytes(15) |> Base.url_encode64
    "[#{time}] #{body}"
  end

  defp poll() do
    interval = :rand.uniform(10) * 1000
    Process.send_after(self(), :write_log, interval)
  end
end
3 Likes