Hello, I have discovered Elixir recently and I have decided to do some projects with it (and Phoenix).

I need to take file from one server and stream it to client through my server. I wanted to do it with streams. My HttpStream looks almost exactly as this: https://labs.civilcode.io/elixir/2018/05/03/stream-files.html.

Streaming works but when I want to access bigger file (for example ~4 GB movie) my application consumes large amounts of memory, and after ~20 seconds video stops playing.

My controller’s function looks like this:

def show(conn, _params) do
    url = "http://localhost:8080/Bigfile.mp4"
    %{headers: proxy_headers} = HTTPoison.head!(url)
    proxy_headers = for {k, v} <- proxy_headers, do: {String.downcase(k), v}, into: %{}

    chunked_conn =
      conn
      |> put_resp_content_type("video/mp4")
      |> put_resp_header("content-length", proxy_headers["content-length"])
      |> send_chunked(200)

    url
    |> HttpStream.stream()
    |> Stream.map(fn n -> chunk(chunked_conn, n) end)
    |> Stream.run()
  end

It’s not beautiful but I’ll improve that later. From my observations chunk/2 is called too fast (if I insert :timer.sleep(1) before stream_next(resp) memory usage is normal but it’s much slower).

This is memory usage from :observer (images are from imgur and they appear to be cropped here):
https://imgur.com/SSOjNie

And very interesting thing is that one process has very long message queue:
https://imgur.com/leANyk4

How to fix this?