Serving a file while it is being created in memory using chunked transfer

Phoenix Forum Questions / Help
, ,
1

Hi,

I am new to elixir and phoenix and I am trying to solve the following problem:

In my application I create an audio file (using a tts system) that takes a while to create (up to several seceonds).

I want http clients (well, the browser) to be apple to get/stream that file:

  • while it is created
  • after it is created

I build a solution for this, but it feels kind of complicated. As I am an absolute beginner and not very experienced on how to build good solutions with elixir, I wonder if someone could give me some input on how to better solve this, or maybe even use existing solutions that I am not aware of. So maybe someone has some input on this?

My solution consists of:

  • an audio file provider, hat creates the audio file form tts
  • an Audio Buffer, that holds the audio created to far in memory. This is implemented as a GenServer.
  • an Audiofile Controller, that works as a phoenix controller and serves the audio file to the browser using chunked transfer

The Process is now like this:

  1. The TTS Creator starts working and sends data to the Audio Buffer GenServer
  2. The Browser requests the file from the Audiofile controller
  3. The controller ask the Audio Buffer for the data so far and …
  4. Receive the buffer with the data so far, which the controller sends to the browser.
  5. If the file is not yet finished, the Audiofile Controller has to receive the remaining data as it is created. For this I use the Phoenix PubSub on the “audiofile” topic, to which the Audiofile Controller subscribes and sends to the Browser the received data to the Browser.
┌───────────────┐    1    ┌──────────────────────┐                   
│               │  send   │                      │   5 send more data
│  TTS Creator  ├─────────►     Audio Buffer     ├───────────┐       
│               │  data   │                      │           │       
└───────────────┘         └────────▲────┬────────┘           │       
                          3 ask for│    │ 4 send        ┌────▼────┐  
                            data   │    │ data buffer   │ PubSub  │  
                       ┌───────────┴────▼─────────┐     └────┬────┘  
                       │                          │          │       
                       │   Audiofile Controller   ◄──────────┘       
                       │                          │                  
                       └─────────────▲────────────┘                  
                                     │2                              
                                     │Download AudioFile             
                              ┌──────┴─────┐                         
                              │            │                         
                              │  Browser   │                         
                              │            │                         
                              └────────────┘

For this I have implemented the AudioController like this (simplified pseudo code):

defmodule MyAppWeb.AudiofileController do
  use MyAppWeb, :controller
  require Logger

  def show(conn, %{"fileid" => file_id}) do
    # make sure we dont miss any messages ...
    MyAppWeb.Endpoint.subscribe("audiofiles")

    case FileStorage.get_file_data(file_id) do
      {:ok, data} ->
        conn = send_chunked(conn, 200)

        send_chunked_file(conn, file_id)
      # other cases
    end
  end

  def send_chunked_file(conn, file_id) do
    receive do
      %Phoenix.Socket.Broadcast{
        topic: "audiofiles",
        event: "filedata",
        payload: {:file_data, name, new_data}
      }
      when name == file_id ->
        conn
        |> chunk(new_data)

        send_chunked_file(conn, file_id)

      %Phoenix.Socket.Broadcast{
        topic: "audiofiles",
        event: "finish",
        payload: {:finish_file, name}
      }
      when name == file_id ->
        # we are done
        conn
    after
      5_000 ->
        Logger.error("Waiting for file #{file_id} timed out")
        conn
    end
  end
end

Especially the fiddling around with the PubSub Messages feels a little, like it could be better. Also this design means, that if multiple files are created in parallel, the PubSub messages from the Audio Buffer reach all Audiofile Controllers, even those serving other files than the one the PubSub message is for.

Does anyone have some Ideas for an more idiomatic way of doing this?

Thank you guys!

2

If you send self() to the function that now broadcasts on the pubsub, you can use Process.send/3 to send back the data, skipping the pubsub altogether and avoid that issue.

1 Like