Read file (csv) uploaded via POST in Plug.Router by only using Plug libraries, no Phoenix awailable

The closest I could get to read the file was seeing it in the body

defmodule Foo.Bar do
  use Plug.Router

  import Plug.Conn
  require Logger

  plug Plug.Logger, log: :debug

  plug :match
  plug :dispatch
  plug(Plug.Parsers, parsers: [:urlencoded, {:multipart, length: 10_000_000}])

  post "/foo/bar/import" do
    Logger.error(inspect conn.params["file"])
    Logger.error(inspect(Plug.Conn.read_body(conn), label: "body"))

And here is the output of Logger:

2023-02-20 20:57:52.722 [error] nil (module=Foo.Bar line=14 )
2023-02-20 20:57:52.740 [error] {:ok, "-----------------------------2239044772062058215875569635\r\nContent-Disposition: form-data; name=\"file\"; filename=\"example.csv\"\r\nContent-Type: text/csv\r\n\r\nh1;h2;h3\nr1c1;r1c2;r1c3\nr2c1;r2c2;r2c3\nr3c1;r3c2;r3c3\n\r\n-----------------------------2239044772062058215875569635--\r\n", %Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{}, body_params: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "", method: "POST", owner: #PID<0.812.0>, params: %{}, path_info: ["foo", "bar", "import"], path_params: %{}, port: 80, private: %{before_send: [#Function<1.3486200/1 in>], plug_route: {"/foo/bar/import", #Function<1.57698496/2 in Foo.Bar.do_match/4>}}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "", remote_ip: {127, 0, 0, 1}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "en-US,en;q=0.5"}, {"authorization", "[[content removed]]"}, {"cache-control", "no-cache"}, {"content-length", "271"}, {"content-type", "multipart/form-data; boundary=---------------------------2239044772062058215875569635"}, {"cookie", "session=1668075015;"}, {"dnt", "1"}, {"host", ""}, {"origin", ""}, {"pragma", "no-cache"}, {"referer", ""}, {"sec-fetch-dest", "iframe"}, {"sec-fetch-mode", "navigate"}, {"sec-fetch-site", "same-origin"}, {"sec-gpc", "1"}, {"te", "trailers"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/109.0"}, {"x-forwarded-for", ""}, {"x-forwarded-host", ""}, {"x-forwarded-port", "443"}, {"x-forwarded-proto", "https"}, {"x-forwarded-server", "root-uld-trk5"}, {"x-real-ip", ""}], request_path: "/foo/bar/import", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :http, script_name: [], secret_key_base: nil, state: :unset, status: nil}} (module=Foo.Bar line=16 )

Unfortunately getting from here to actually read the file part of the body got me stuck, found no valid example, tried many things I found online that seem to be deprecated as no longer available in Plug 1.14, like get_upload/2, Plug.Upload.parse_multipart/2, get_multipart_params/2, Plug.Upload.get_params/1, Plug.Upload.get/2.

Tried to get_req_header and if it’s “multipart/form-data” do Plug.Multipart.parse or Plug.Parsers.MULTIPART.parse/5 that’s deprecated I guess or I failed to understand how to use, then trying with a recursive with Plug.Parsers.MULTIPART.init([]) and Plug.Parsers.MULTIPART.parse/5

I am definitely miss something here, anyone has a valid example E2E on how can I read the file in the Plug above by only using plug, where the files is clearly visible in the body that’s “multipart/form-data” (P.S. I can’t control the requester to change the content-type).

Thank you

Looks like you’re close. You should try to read the form parameter that the file is uploaded to.

If that’s not enough I’d suggest posting a more complete code example so we can engage with it and try and fix it.

How are you producing this request? The output you posted shows the body using a different boundary than the Content-Type specifies:

# from the body
# from the Content-type - not enough -s

The multipart/mixed spec requires that a boundary delimiter appear at the beginning of a line, so this message isn’t being successfully parsed.

That is a very good observation and I see this in the browser console, no clue why it is like this, I know that PHP back-end that is currently used to receive the file is not affected by it (that’s why maybe it was not observed so far).

The request is produced in the browser by an antique UI (almost 15y old written in ExtJs 2, legacy at it’s “best”)

I was not sure where to post more code, so I made a minimum project that reveals this, the code is in api.ex where you can also see lots of things commented out from things I tried.
I also attached a har file as exported from the browser when doing the upload, and the body of the request as I see in the logs if I read it via Plug.Conn.read_body(conn), label: "body")

If there are other suggestions on where to post the code to be easier please let me know.
Thank you

1 Like

You can just make a public GitHub repository?

Good suggestion, here it is

Apologies - I misread RFC2046 and completely missed the part where the lines separating the parts are -- followed by the boundary! The request is correctly formed.

The issue is simpler: Plug.Parser isn’t setting params because it isn’t being called, because it’s listed after plug :dispatch.

Thank you, swapping the Parsers before the others, like this:

  plug(Plug.Parsers, parsers: [:urlencoded, {:multipart, length: 10_000_000}])
  plug :match
  plug :dispatch

makes the %Plug.Upload{} available and filled with the correct info inside the conn.params["file"] now.