How can I send a chunked file to a user for download?

shahryarjb · July 5, 2019, 8:07pm

Hello, I created a post (https://elixirforum.com/t/downloading-with-user-token) before and I need a Feature which helps me to send a file that be chunked.
Imagine I have big video and my user should download it, I need to chunk this file, mean this file should be cut to multi parts (chunk).

normal Phoenix download

  def download(conn, _params) do
    file = File.read!("/Users/test/name.png")

    conn
     |> put_resp_content_type("image/png")
     |> put_resp_header("Content-disposition","attachment; filename=\"test.png\"")
     |> put_resp_header("X-Accel-Redirect", "/tempfile/download/test.png")
     |> put_resp_header("Content-Type", "application/octet-stream")
     |> send_resp(200, file)
  end

this is not a stream file, if I change the first line to this:

file = File.stream!("/Users/shahryar/Desktop/Khat-Ghalam.png", [], 204800)

I have this error:

no function clause matching in Plug.Conn.resp/3

if I use |> send_resp(200, file.path) it works sometimes . but I think it isn’t a stream file and I can’t create it many parts.
it should be noted, I wanted to use |> send_chunked(200) in my phoenix but I have error.

at least I need to create a download file that sends the file as section to section to my user

Thanks

hubertlepicki · July 5, 2019, 8:27pm

You probably want to avoid reading file to memory as you ready figured. You can probably use Plug.Conn.send_file/5.

shahryarjb · July 5, 2019, 9:24pm

would you mind giving me a sample code ? this Plug.Conn.send_file/5. can send stream chunked file ?

benwilson512 · July 5, 2019, 9:56pm

@shahryarjb as I understand it you don’t need to manually make it stream, it always streams. There is an example in the documentation: https://hexdocs.pm/plug/1.8.1/Plug.Conn.html#send_file/5-examples

shahryarjb · July 6, 2019, 6:49am

Hello @benwilson512, Thanks,
You are thinking it doesn’t matter I want to create my custom chunk? please see this code, it uses streams, I can’t create my custom chunk yet but it uses it and do more job which I don’t need I mean it downloads file on my server, but we could just show file path instead of downloading .

  def download(conn, _params) do
    url = "http://localhost:4000/images/logo.png"
    %HTTPoison.AsyncResponse{id: id} = HTTPoison.get!(url, %{}, stream_to: self())

    conn = conn
    |> put_resp_content_type("image/event-stream")
    |> put_resp_header("Content-disposition","attachment; filename=\"test.png\"")
    |> put_resp_header("X-Accel-Redirect", "/tempfile/download/test.png")
    |> put_resp_header("Content-Type", "application/octet-stream")
    |> send_chunked(200)

    process_httpoison_chunks(conn, id)
  end

  def process_httpoison_chunks(conn, id) do
    receive do
      %HTTPoison.AsyncStatus{id: ^id} ->
        # TODO handle status
        process_httpoison_chunks(conn, id)
      %HTTPoison.AsyncHeaders{id: ^id, headers: %{"Connection" => "keep-alive"}} ->
        # TODO handle headers
        process_httpoison_chunks(conn, id)
      %HTTPoison.AsyncChunk{id: ^id, chunk: chunk_data} ->
        conn |> chunk(chunk_data)
        process_httpoison_chunks(conn, id)
      %HTTPoison.AsyncEnd{id: ^id} ->
        conn
    end
  end

reference : logging - How to read an HTTP chunked response and send chunked response to client in Elixir? - Stack Overflow

in elixir doc

Sends a file as the response body with the given status and optionally starting at the given offset until > the given length.

If available, the file is sent directly over the socket using the operating system sendfile operation.

It expects a connection that has not been :sent yet and sets its state to :file afterwards. Otherwise raises Plug.Conn.AlreadySentError .

I read this but I couldn’t find the place that tells me it is a stream sending file.

Thank you

NobbZ · July 6, 2019, 7:45am

Because it might be, that it is not even a Stream (in elixirs understanding of the word). The documentation claims to use the kernels sendfile if available. In theory the most efficient way to get the file from disk to socket, as this is a kernel native operation.

The BEAM process that handles your request, wont get scheduled anymore until the sendfile has been finished.

hubertlepicki · July 6, 2019, 8:24am

Well, I don’t know what you mean precisely here or that you know what “chunked response” is in HTTP. Judging what you want to achieve, i.e. send video file to client / player, you most likely want to stream the file.

Sending file to client with send_file will, most likely, meet your requirements. It instructs Cowboy to send the file to client, and it will be done in efficient manner, reading parts of the file from disk and sending it to the client as it receives previous parts. This should be good enough to do basic video player for example.

This may not be good enough if you want to do some smart buffering, i.e. want to load just part of the file to buffer next 15s and then when user plays some more, buffer another 15s. You would have to do some sync with player and control the streaming process yourself, and yes in such case probably relying on send_file will not be enough. But for basic streaming it will be enough and efficient enough.

shahryarjb · July 6, 2019, 11:52am

I am very grateful

Pleas see my code:

file_streamed = File.stream!("/Khat-Ghalam.png", [], 200)
    conn = conn
    |> put_resp_content_type("image/event-stream")
    |> put_resp_header("Content-disposition","attachment; filename=\"test.png\"")
    |> put_resp_header("X-Accel-Redirect", "/tempfile/download/test.png")
    |> put_resp_header("Content-Type", "application/octet-stream")
    |> send_chunked(200)

    file_streamed
    |> Enum.with_index
    |> Enum.each(fn {content, index} ->
      conn |> chunk(content)
    end)

this code works me and lets me create chunked file with size that I want. I just tested it with png file. I understand phoenix handle my request but it is auto, not with custom chunked file size. now my code has a problem in phoenix controller

this error I just need to fix it if my way is true

[error] #PID<0.3708.0> running KhatoghalamWeb.Endpoint (connection #PID<0.3707.0>, stream id 1) terminated
Server: localhost:4000 (http)
Request: GET /download
** (exit) an exception was raised:
    ** (RuntimeError) expected action/2 to return a Plug.Conn, all plugs must receive a connection (conn) and return a connection, got: :ok
        (khatoghalam) lib/khatoghalam_web/controllers/client_home_controller.ex:1: KhatoghalamWeb.ClientHomeController.phoenix_controller_pipeline/2

wanton7 · July 6, 2019, 12:01pm

There bug in Cowboy 2.x (no back pressure). That means chunked upload will use lot of memory, basically load file in to memory first. Very high memory usage when streaming file with Phoenix

shahryarjb · July 6, 2019, 1:09pm

Yes, I need more than now , for example we want to create a video streaming or file, after each activity I want to send my user a part of chunked files. but now I can just send all the file at first and I don’t want it.

Imagine you are in the game and I want to send u a gift after each win a request.

win 1 = chunked_file_content_1
win 2 = chunked_file_content_2
win 3 = chunked_file_content_3
win 4 = chunked_file_content_4

hubertlepicki · July 6, 2019, 1:15pm

I think you need to return conn from the funciton and you may not be returning it?

shahryarjb · July 6, 2019, 1:17pm

I don’t know where I should return conn

my controller function:

  def download(conn, _params) do
    file_streamed = File.stream!("/Khat-Ghalam.png", [], 200)
    conn = conn
    |> put_resp_content_type("image/event-stream")
    |> put_resp_header("Content-disposition","attachment; filename=\"test.png\"")
    |> put_resp_header("X-Accel-Redirect", "/tempfile/download/test.png")
    |> put_resp_header("Content-Type", "application/octet-stream")
    |> send_chunked(200)

    file_streamed
    |> Enum.with_index
    |> Enum.each(fn {content, _index} ->
      conn |> chunk(content)
    end)
  end

hubertlepicki · July 6, 2019, 1:21pm

at the last line. You are not returning conn. Put “conn” just before “end”.

NobbZ · July 6, 2019, 1:23pm

Your Enum.each returns :ok, you need to return a Plug.Conn.t().

Easiest is to use Enum.reduce/3, similar to how Enum.reduce_while/3 is used in the docs for Plug.Conn.chunk/2…

|> Enum.reduce(conn, {content, _}, conn -> # why the with_index at all when ignoring the index?
  chunk(conn, content)
end

NobbZ · July 6, 2019, 1:27pm

Simply returning an “old” conn at the end of the function won’t probably work as its “meta data” hasn’t been updated, its best to return the conn as it was returned by Plug.Conn.chunk/2 to be sure all metadata got updated.

shahryarjb · July 6, 2019, 1:27pm

because I just show you the example, I use it and count all the content and save it to my db, but I didn’t know these Enum.reduce/3 I will test

NobbZ · July 6, 2019, 1:30pm

Its probably the most important function in Enum. Any other can be implemented on top if this.

shahryarjb · July 6, 2019, 1:30pm

I call conn after end it works, please see this pic

58%20pm

how can do ?

shahryarjb · July 6, 2019, 1:37pm

is my user waiting next part of chunked file after a time ?

I test it:

file_streamed
    |> Enum.with_index
    |> Enum.each(fn {content, _index} ->
      conn |> chunk(content)
      Process.sleep 2000
    end)

I have The network connection was lost. error, now there is a problem more , I should find a way that tells the user client download manager , Wait to finish next request

NobbZ · July 6, 2019, 1:46pm

The client will ever try to receive as fast as it can.

Unless you tell it to do otherwise.

A common way to do streaming media is not to send a chunked response, but to request individual chunks on the same keep-alived connection.