Phoenix w/ broken HTTP client: Upload w/o Content-Length or transfer-encoding header

Phoenix Forum Questions / Help
1

Hi everybody,

I’m writing a Phoenix app where one client for a specific endpoint is basically broken. It’s neither including a transfer-encoding: chunked nor a Content-Length: … header while it’s uploading data. The client is ubiquitous and I don’t control it. There is no chance I can get the client fixed.

I successfully simulated the broken client with curl. Here are the symptoms. curl works fine with transfer-encoding: chunked:

curl -v -i --no-buffer -XPUT -H 'Content-Length:' -H 'Expect:' -H 'transfer-encoding: chunked' --data-binary '@somefile' http://localhost:4000/foo

So basically I’m forcing curl to drop the Content-Length: … header (and the “Expect: 100-continue”). But I’m including the transfer-encoding: chunked header. This works well.

But this doesn’t

curl -v -i --no-buffer -XPUT -H 'Content-Length:' -H 'Expect:' -H 'transfer-encoding:' --data-binary '@somefile' http://localhost:4000/foo

Now I’m also dropping the ‘transfer-encoding: chunked’. I’m - rightfully - getting a 505 HTTP Version Not Supported. And in the app read_body/2 gets a {:ok, body, conn} tuple where the body is an empty string.

I tried to just use a plug which adds a transfer-encoding: chunked header in the endpoint.ex:

defmodule StreamerWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :streamer

  socket "/socket", StreamerWeb.UserSocket

  def add_transfer_encoding(conn, _) do
    update_req_header(conn, "transfer-enconding", "chunked", &(&1))
  end
  plug :add_transfer_encoding
  […]

which makes ‘transfer-encoding: chunked’ appear in my controller, but the body still isn’t readable. I suspect that by the time I set the header it’s already to late.

What can I try to make this work? Any help is highly appreciated.

Niko.

2

Update:

I’ve learned that transfer-encoding was only introduced with HTTP/1.1 and [Content-Length: …] is mandatory for HTTP/1.0 for requests with a body.

Also it seems that cowboy doesn’t support reading bodies from requests that miss both [transfer-encoding] and [Content-Length: …] headers.

As the shitty client I have to support uses (basically) HTTP/1.0 to stream content to the server and thus the content length is unknown beforehand they decided to skip the Content-Length: … header. And transfer-encoding wasn’t invented yet.

I can see two possibilities:

  • Hack cowboy to not disregard the body although none of the two headers is present.
  • Add a proxy (e.g. HAProxy) to the stack to add a header.

I’d prefer the first. Does anyone know how this could be done?

Niko.

1 Like
4

I’d probably just open a tcp connection and handle it all manually. Erlang’s :tcp has a packet type for it as well too that might help.

5

^ @Gazler

6

Thanks. I got the basics working with a simple ranch tcp server. I’m not to keen to do manual HTTP parsing. Never done that before but my impression is it’s a mine field.

7

Hence why using Erlang’s built-in http parser in the tcp stack is useful. :slight_smile:

8

Oh. Sounds good. Any hints on where I could find documentation or an example?

9

Standard built-in :gen_tcp docs: http://erlang.org/doc/man/gen_tcp.html

Just make it like any other normal :gen_tcp server and specify any of {:packet, :http}, {:packet, :httph}, {:packet, :http_bin}, or {:packet, :httph_bin} depending on how you want to receive the parsed data.

10

Thanks. I started looking into that.

Meanwhile getting support for cowboy reading bodies of such broken requests does not seem out of question: https://github.com/ninenines/cowboy/issues/1212 :smile:. If this plays out, I’d prefer that solution.

1 Like