Very high memory usage when streaming file with Phoenix

Only from reading roughly your post, not the linked code… But it seems as if data comes faster as you are able/willing to process it.

Perhaps you need some rate limiting on the reading end? But probably you need to change a lot of code to do so.

My bad, here is the code: GitHub - Arquanite/phoenix-streaming-app: This example code shows big memory requirements of the application :( · GitHub

I hope there is a way to get more data from original server only after data is sent to client (normal http servers must be working similar way?).

╰─➤  /usr/bin/time -f "mem=%K RSS=%M elapsed=%E cpu.sys=%S user=%U" -- mix test 
..

Finished in 0.08 seconds
2 tests, 0 failures

Randomized with seed 92121
mem=0 RSS=52428 elapsed=0:01.09 cpu.sys=0.28 user=1.10

I’m running your tests but I’m not seeing high memory usage here. What’s the specific test to run that shows the problem?

I’m still thinking the issue is just what @NobbZ said, that the data is being received and stored without being throttled faster than it can resend it out.

There’s no tests for this (I don’t have much experience in testing and don’t know how to test this without uploading really big binary somewhere).

I’m testing this this way:
http-server (from npm) is listening on port 8080 (and serving my video file named “Bigfile.mp4”). Then I open my browser on localhost:4000/api/files/1 (id doesn’t matter) and it starts loading video.

As you can see there is really small amount of code so I can make something new that will be more efficient. But I don’t know how.

Ah, no problem then. And to get a ‘big infinite set of data’ can just read /dev/random or so. ^.^

I took a look and it seems you are using HTTPoison, which uses hackney behind it, and as I recall it does use active: :once on the TCP stack when you pass in active: :once to HTTPoison, so that should be fine… Maybe it’s some growing memory somewhere rather than something actually being stored…

At this point I’d really use :observer to see which process is allocating that memory then run a GC on that process, if that lowers the memory then it’s just unused memory that hasn’t used enough of the system memory to cause a GC within it’s time yet (and there are a few fixes for this, but eh). If it’s actually allocated memory though then something is holding on to it, which could be hackney, httpoison, or Stream from what I see in your code, and I doubt it would be Stream. Let’s look at the consumer perhaps…

Hmm, in your controller:

    chunked_conn =                                                                    
      conn                                                                            
      |> put_resp_content_type("video/mp4")                                           
      |> put_resp_header("content-length", proxy_headers["content-length"])           
      |> send_chunked(200)                                                            
                                                                                      
    url                                                                               
    |> HttpStream.stream()                                                            
    |> Stream.map(fn n -> chunk(chunked_conn, n) end)                                 
    |> Stream.run()                                                                   
  end

Hmm, as I recall the response body get’s accumulated in the conn.resp_body, but that’s not being accumulated here. I do know that conn can be used as an ‘into’ so the whole streaming part could be replaced with:

    url
    |> HttpStream.stream()
    |> Stream.into(chunked_conn)
    |> Stream.run()

However I think that might accumulate the body, not sure…

I haven’t used chunks in plug yet outside of trivial things… Hmm…

Oh wait!!!

It’s Stream.map! It’s storing all past sent data!

Just replace Stream.map in your existing code with Stream.each so it doesn’t save the result. (Stream.into might work too? If it doesn’t accumulate, I’m not sure)

HTTPoison is doing good job, when i replace chunk with other code the memory usage is fine.
I used :observer and it shows which process is consuming memory (screen is in first post). It’s something related to cowboy. I’m really a beginner but I think the problem is with this long message queue (and gc won’t help with it?).

And Stream.each does not improve anything :<

Well you definitely want each instead of map there as map would have included all past sent data, so… ^.^;

If ‘each’ doesn’t help, maybe try into(chunked_conn)? I’m really surprised each didn’t fix it…

Still nothing, it loads whole file into memory (it uses 4.6 GB of RAM, almost exact size of my video)

EDIT: Maybe I should use something else than conn, but what and how integrate it into Phoenix app?

You should change your controller function to return conn. Now it returns whatever Stream.run() is returning. So change part of your code to

    conn =
      conn
      |> put_resp_content_type("video/mp4")
      |> put_resp_header("content-length", proxy_headers["content-length"])
      |> send_chunked(200)

    url
    |> HttpStream.stream()
    |> Stream.each(fn n -> chunk(conn, n) end)
    |> Stream.run()

    conn

Not sure it will help with memory problem, but every controller function needs to return Plug.Conn.t()