NGINX doesn't let phoenix to send file to it's user

shahryarjb · August 9, 2019, 12:22pm

hello, I create these post before and made my code in localhost that works for me perfect ,

https://elixirforum.com/t/downloading-with-user-token/

https://elixirforum.com/t/how-can-i-send-a-chunked-file-to-a-user-for-download

but today I decided to move my project into my server, the server uses NGINX and docker but doesn’t let me download with Phoenix, Please see my code!!

my code:

  def download(conn, file_path, file_type, file_name, "normal") do
    file = File.read!("#{file_path}")

    conn
     |> put_resp_content_type("#{file_type}")
     |> put_resp_header("Content-disposition","attachment; filename=\"#{file_name}#{Path.extname(file_path)}\"")
     |> put_resp_header("X-Accel-Redirect", "/tempfile/download/#{file_name}#{Path.extname(file_path)}")
     |> put_resp_header("Content-Type", "application/octet-stream")
     |> send_resp(200, file)
  end

I have tested 2 file format and size, the lowest size and big file, but they were different.

when I tested big file I had this error just in my safari that my browser download html file and tells me

no route found for GET /lms/download/c416ed0c-67d3-455c-b6f3-77273228be22.mp4 (KhatoghalamWeb.Router)

but the low size shows me a nginix default error

404 Not Found

nginx

you can see these link online: http://bit.ly/31yFISx

Middle of the page you can see this image and click on download icon

chrome error:

This site can’t be reached The web page at https://newkhat.khatoghalam.com/lms/download?chapter_id=5e638cce-e6b0-4d14-89f8-1f754c191ea9&type=رایگان might be temporarily down or it may have moved permanently to a new web address.
ERR_INVALID_RESPONSE

how can I fix this ?

Update

Big file download like video iex console on server:

[info] GET /lms/download
[debug] Processing with KhatoghalamWeb.ClientLmsController.download/2
  Parameters: %{"chapter_id" => "5e638cce-e6b0-4d14-89f8-1f754c191ea9", "type" => "رایگان"}
  Pipelines: [:browser, :csrf]
[debug] QUERY OK source="lms_chapters" db=1.9ms queue=0.3ms
SELECT l0."id", l1."id", l1."course_id", l0."download_link", l0."inserted_at", l0."updated_at" FROM "lms_chapters" AS l0 INNER JOIN "lms_headlines" AS l1 ON l1."id" = l0."headline_id" WHERE (l0."id" = $1) AND (((l0."status" = TRUE) AND (l1."status" = TRUE)) AND (l0."type" = 'رایگان')) [<<94, 99, 140, 206, 230, 176, 77, 20, 137, 248, 31, 117, 76, 25, 30, 169>>]
[debug] QUERY OK source="lms_courses" db=2.1ms queue=0.1ms
SELECT l0."id" FROM "lms_courses" AS l0 INNER JOIN "lms_categories" AS l1 ON l1."id" = l0."category_id" WHERE (l0."id" = $1) AND ((l0."status" = TRUE) AND (l1."status" = TRUE)) [<<57, 222, 97, 78, 28, 189, 70, 214, 160, 161, 15, 29, 94, 185, 131, 112>>]

[info] Sent 200 in 9ms
[info] GET /tempfile/download/5e638cce-e6b0-4d14-89f8-1f754c191ea9.mp4
[debug] ** (Phoenix.Router.NoRouteError) no route found for GET /tempfile/download/5e638cce-e6b0-4d14-89f8-1f754c191ea9.mp4 (KhatoghalamWeb.Router)
    (khatoghalam) lib/phoenix/router.ex:316: KhatoghalamWeb.Router.call/2
    (khatoghalam) lib/khatoghalam_web/endpoint.ex:1: KhatoghalamWeb.Endpoint.plug_builder_call/2
    (khatoghalam) lib/plug/debugger.ex:122: KhatoghalamWeb.Endpoint."call (overridable 3)"/2
    (khatoghalam) lib/khatoghalam_web/endpoint.ex:1: KhatoghalamWeb.Endpoint.call/2
    (phoenix) lib/phoenix/endpoint/cowboy2_handler.ex:33: Phoenix.Endpoint.Cowboy2Handler.init/2
    (cowboy) /khatogh/deps/cowboy/src/cowboy_handler.erl:41: :cowboy_handler.execute/2
    (cowboy) /khatogh/deps/cowboy/src/cowboy_stream_h.erl:296: :cowboy_stream_h.execute/3
    (cowboy) /khatogh/deps/cowboy/src/cowboy_stream_h.erl:274: :cowboy_stream_h.request_process/3
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

low file size in server iex console

[info] GET /lms/download
[debug] Processing with KhatoghalamWeb.ClientLmsController.download/2
  Parameters: %{"chapter_id" => "c416ed0c-67d3-455c-b6f3-77273228be22", "type" => "رایگان"}
  Pipelines: [:browser, :csrf]
[debug] QUERY OK source="lms_chapters" db=2.0ms queue=0.7ms
SELECT l0."id", l1."id", l1."course_id", l0."download_link", l0."inserted_at", l0."updated_at" FROM "lms_chapters" AS l0 INNER JOIN "lms_headlines" AS l1 ON l1."id" = l0."headline_id" WHERE (l0."id" = $1) AND (((l0."status" = TRUE) AND (l1."status" = TRUE)) AND (l0."type" = 'رایگان')) [<<196, 22, 237, 12, 103, 211, 69, 92, 182, 243, 119, 39, 50, 40, 190, 34>>]
[debug] QUERY OK source="lms_courses" db=1.2ms queue=0.9ms
SELECT l0."id" FROM "lms_courses" AS l0 INNER JOIN "lms_categories" AS l1 ON l1."id" = l0."category_id" WHERE (l0."id" = $1) AND ((l0."status" = TRUE) AND (l1."status" = TRUE)) [<<57, 222, 97, 78, 28, 189, 70, 214, 160, 161, 15, 29, 94, 185, 131, 112>>]

NobbZ · August 9, 2019, 1:13pm

How does your routes look like?

shahryarjb · August 9, 2019, 1:20pm

Hello,

this line:

|> put_resp_header("X-Accel-Redirect", "/tempfile/download/#{file_name}#{Path.extname(file_path)}")

tempfile/download is a fake address, my user sends me request that has a course id like this:

http://localhost:4000/lms/download?chapter_id=2e0b99dd-9219-4752-8007-4eeefcbff6ca&type=%D8%A7%D8%B4%D8%AA%D8%B1%D8%A7%DA%A9%DB%8C

and this router is:

get "/lms/download", ClientLmsController, :download

that’s it, I have no router for fake address in download header

     |> put_resp_content_type("#{file_type}")
     |> put_resp_header("Content-disposition","attachment; filename=\"#{file_name}#{Path.extname(file_path)}\"")
     |> put_resp_header("X-Accel-Redirect", "/tempfile/download/#{file_name}#{Path.extname(file_path)}")
     |> put_resp_header("Content-Type", "application/octet-stream")

it should be noted, it works in my localhost default Phoenix server
Thanks

NobbZ · August 9, 2019, 2:02pm

What’s your route for the /tempfile/...? You are redirecting there and therefore that’s what the client will request eventually.

shahryarjb · August 9, 2019, 2:16pm

I have no router for /tempfile/...?, because don’t know what it should do? what does this router run?

I don’t want to redirect, just wanted to show a fake address, what is your suggestion?

|> put_resp_header("X-Accel-Redirect", "/tempfile/download/#{file_name}#{Path.extname(file_path)}")

by the way I have edited /tempfile/download to lms/download, but it wasnt fixed.

my controller `/lms/download`

def download(conn, %{"chapter_id" => chapter_id, "type" => "رایگان"}) do
    with {:ok, :chapter, chapter_info} <- LmsQuery.chapter_select_download(chapter_id, "رایگان"),
         {:ok, :course, _course_info} <- LmsQuery.course(chapter_info.course_id, "select") do


        Khatoghalam.Extera.Download.download(conn, "/Users/sec_file/#{chapter_info.download_link}", "video/mp4", chapter_info.id, "normal")
    else
      _ ->
        conn
        |> put_flash(:error, "error")
        |> redirect(to: "#{KhatoghalamWeb.Router.Helpers.client_lms_path(conn, :home)}")
    end
  end

  def download(conn, file_path, file_type, file_name, "normal") do
    file = File.read!("#{file_path}")

    conn
     |> put_resp_content_type("#{file_type}")
     |> put_resp_header("Content-disposition","attachment; filename=\"#{file_name}#{Path.extname(file_path)}\"")
     |> put_resp_header("X-Accel-Redirect", "/tempfile/download/#{file_name}#{Path.extname(file_path)}")
     |> put_resp_header("Content-Type", "application/octet-stream")
     |> send_resp(200, file)
  end

that code is all of my code for downloading a file. how can I handle fake pass or a router ?

|> put_resp_header("X-Accel-Redirect", "/tempfile/download/#{file_name}#{Path.extname(file_path)}")

I didnt understand how could I fix it ? would you mind giving a router or example for fixing this? please

alvises · August 9, 2019, 2:55pm

Can you please show us the router.ex file you should find at lib/khataghalam_web/router.ex of your Phoenix project?

shahryarjb · August 9, 2019, 2:57pm

scope "/", KhatoghalamWeb do
  pipe_through [:browser, :csrf]

  get "/lms/course/:alias_link", ClientLmsController, :course
  get "/lms/download", ClientLmsController, :download
end

Thanks

OvermindDL1 · August 9, 2019, 3:08pm

You might be sending too much data in one burst since you say it only doesn’t work on “big files”.

This is loading the entire big file into active memory, yet you aren’t doing anything to it, unsure why?

Then you are sending the file out in one big message instead of streaming it out.

You should be using send_file/5 instead, this will stream it out, not eating RAM, and will naturally throttle to how fast the proxy will send it. Or you can chunk it manually but that’s a lot of work for no gain. ^.^;

If you switch to that though, does it work now?

OvermindDL1 · August 9, 2019, 3:13pm

And I have no clue what this done, never ran across that before… But it is definitely significantly non-standard.

shahryarjb · August 9, 2019, 3:22pm

no im not sure, I took this code on those link in first post.

but when I use this:

  def download(conn, file_path, file_type, file_name, "normal") do
    file = File.read!("#{file_path}")

    conn
     |> put_resp_content_type("#{file_type}")
     |> put_resp_header("Content-disposition","attachment; filename=\"#{file_name}#{Path.extname(file_path)}\"")
     |> put_resp_header("X-Accel-Redirect", "/tempfile/download/#{file_name}#{Path.extname(file_path)}")
     |> put_resp_header("Content-Type", "application/octet-stream")
     |> send_file(200, file)
  end

I have this error:

no function clause matching in Plug.Conn.send_file/5

shahryarjb · August 9, 2019, 3:29pm

I made mistake this works in localhost now I should test it on server

OvermindDL1 · August 9, 2019, 3:35pm

You aren’t sending the file, you are sending the contents, your code should be like:

  def download(conn, file_path, file_type, file_name, "normal") do
    conn
     |> put_resp_content_type("#{file_type}")
     |> put_resp_header("Content-disposition","attachment; filename=\"#{file_name}#{Path.extname(file_path)}\"")
     |> put_resp_header("Content-Type", "application/octet-stream")
     |> send_file(200, "#{file_path}")
  end

It takes a filename, it can’t stream an existing body. send_file/5 works by asking the kernel to send the file contents to the socket, it never enters userspace at all, no faster method of sending a file (in general, in reality it chunks it in certain modes, streams it in others, etc… it tries to do whatever is fastest and most efficient without overwhelming things).

shahryarjb · August 9, 2019, 3:58pm

Thank you it works ,

I have tested a streaming way, but it wasn’t successful, in this way I cant create a custom chunk what I need.

def download(conn, file_url, file_type, file_name, "stream") do

    %HTTPoison.AsyncResponse{id: id} = HTTPoison.get!(file_url, %{}, stream_to: self())
    conn = conn
    |> put_resp_content_type("#{file_type}")
    |> put_resp_header("Content-disposition","attachment; filename=\"#{file_name}#{Path.extname(file_url)}\"")
    |> put_resp_header("Content-Type", "application/octet-stream")
    |> send_chunked(200)
    # image/event-stream
    process_httpoison_chunks(conn, id)
  end

  def process_httpoison_chunks(conn, id) do
    receive do
      %HTTPoison.AsyncStatus{id: ^id} ->
        # TODO handle status
        process_httpoison_chunks(conn, id)
      %HTTPoison.AsyncHeaders{id: ^id, headers: %{"Connection" => "keep-alive"}} ->
        # TODO handle headers
        process_httpoison_chunks(conn, id)
      %HTTPoison.AsyncChunk{id: ^id, chunk: chunk_data} ->
        conn |> chunk(chunk_data)
        process_httpoison_chunks(conn, id)
      %HTTPoison.AsyncEnd{id: ^id} ->
        conn
    end
  end

end

OvermindDL1 · August 9, 2019, 4:01pm

What error were you having with that? (Maybe in a new thread)

shahryarjb · August 9, 2019, 4:04pm

I have no error, but I cant create my chunk size and sometimes the file user downloads after finishing, the file which downloaded is Broken

shahryarjb · August 9, 2019, 4:06pm

OvermindDL1 · August 9, 2019, 4:13pm

Huh?

<picture>

You don’t set a chunk size for the other server to send to you, you don’t control that as a client. You combine it yourself and handle it however you want otherwise.

shahryarjb · August 9, 2019, 7:37pm

I thought I could read file with File.Stream and then make the file Streamed to divide many chunk size what I need like 200kb after all Send to my user, I was wrong, I wanted to force my user to download chunked file

Thank you , it was a big problem for me.

NGINX doesn't let phoenix to send file to it's user

Update

my controller /lms/download

my controller `/lms/download`