[Phoenix] content-encoding gzip auto deflate/gunzip body on incoming requests

I would like to be avoid to receive content-encoding gzip and have transparent gzip/deflate when content reach my controller.

Using
phoenix 1.5.0
elixir 1.10.2
erlang 22.3
jason 1.0
plug_cowboy 2.0
tesla 1.3

Here is some code to show you what I see:

Client side:

defmodule Client do
  use Tesla

  plug(Tesla.Middleware.JSON)
  plug(Tesla.Middleware.Compression)

  def post, do: post("localhost:4000", %{hello: :i_m_fine})
end

Server side:

defmodule Router do
  use MyAppWeb, :router

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/test_notification" do
    pipe_through :api
    post("/", MyAppWeb.TestController, :test)
  end
end

defmodule MyAppWeb.TestController do
  @moduledoc false
  use MyAppWeb, :controller

  def test(conn, _args) do
    conn
    |> send_resp(200, "yeah me too !")
    |> halt()
  end
end

Doing a request:

# in my conn headers
req_headers: [
  {"connection", "keep-alive"},
  {"content-encoding", "gzip"},
  {"content-length", "123"},
  {"content-type", "application/json"},
  {"host", "localhost:4000"}
]

# fail before reaching my controller, fail in Plug.Parser that expect to Json decode

** (Plug.Parsers.ParseError) malformed request, a Jason.DecodeError exception was raised with message "unexpected byte at position 0: 0x1F"
    (plug 1.10.1) lib/plug/parsers/json.ex:88: Plug.Parsers.JSON.decode/2
    (plug 1.10.1) lib/plug/parsers.ex:313: Plug.Parsers.reduce/8
    (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.plug_builder_call/2
    (my_app 0.1.0) lib/plug/debugger.ex:132: MyAppWeb.Endpoint."call (overridable 3)"/2
    (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.call/2
    (phoenix 1.5.3) lib/phoenix/endpoint/cowboy2_handler.ex:65: Phoenix.Endpoint.Cowboy2Handler.init/4

I could add my own little plug in Enpoint.ex that does the feature, but I feel like I miss an obvious option to configure my phoenix endpoint.

Have a nice day everyone !

1 Like

isnt this something in the lines of:

config :foo, FooWeb.Endpoint,
  http: [
    compress: true,
    protocol_options: [max_keepalive: 5_000_000]
  ],

(The compress: true part?)

Also it looks like compress: true is to compress response from server, not about incomming zipped body.

@rjk Tried it, same behaviour, here is what I have in my Endpoint conf:

iex(3)> Application.get_env(:my_app, MyappWeb.Endpoint)

[
  url: [host: "localhost"],
  secret_key_base: "******",
  render_errors: [view: MyappWeb.ErrorView, accepts: ["json"], layout: false],
  pubsub_server: Myapp.PubSub,
  live_view: [signing_salt: "****"],
  http: [port: 4000, compress: true],
  debug_errors: true,
  code_reloader: true,
  check_origin: false,
  watchers: []
]

Is the request containing a file that needs to be compressed, or you’re after the whole HTTP request being gzipped when being sent from the client?

you’re completely right, I responded too fast.

Context around this is that there are still open tickets/request in cowboy(http server) and Plug about this, the summary is that it’s a bit tricky because of zip bombs (AFAICS). But this should not stop you for your own API.

How I would go about this for now:

  • Remove compression from the client calls (comment that tesla plug in the client)
  • I would directly point your HTTP client calls to your controller (removing the JSON parser pipeline part)
    so now you get the client request ‘raw’ into your controller
  • now JSON decode it yourself inside that controller/action
  • if that works now add compression back in (uncomment that tesla compression plug again in that client)
  • now i expect the same error again as before about ‘cannot parse json’ (but now you can fix it completely inside that controller action itself)
  • add in the gunzipping of the body in the same way Tesla does it, see here (https://github.com/teamon/tesla/blob/v1.3.3/lib/tesla/middleware/compression.ex#L67) so you’ll have something along the lines of :zlib.gunzip body of the request inside your controller action.

now you have it working and can refactor it with; create a Plug (you can add a plug folder in the same folder as where controllers/templates/views folders are located) and now in the call action from your plug you only add back in the gunzipping part. Now you can go back to your old code where you had it going through that API pipeline with json plug but put that gunzip plug before it.

Seems a bit of work but will be worth it in the end :slight_smile:

ps. please be aware that if you do larger multipart/‘streaming’ posts from the client you have to handle it a bit differently and this could surprise you.

ps2. If it’s now working out for you, try to publish your code into the open (e.g. github) as a minimal example because then we can help you way better and respond with something which is directly usable in form of code commits / verified to work :slight_smile:

@dimitarvp the request I’m receiving on my api contain a ziped json as a body, with following header:

{“content-encoding”, “gzip”},
{“content-type”, “application/json”}

@rjk: of course I can’t controller the client side, like you say the solution would be to remove the Plug.Parsers (JSON decode) from Endpoint.ex and handeling it myself.
I would be ideal to create a new plug that I would be before Plug.Parsers that gunzip if needed according to conn header. I started doing that but I have trouble to set to body back in the conn, it looks like it’s not to correct way of doing it.

I’ve created https://github.com/xward/phoenix_fail_to_process_ziped_body for you, will be very easy to reproduce (cf readme.md)

Thanks !

Ok got it working, the JSON decoding part is ‘lower’ in the plug pipeline than we thought, you can see it in your endpoint in the parsing bit. That’s also the place where I got it working (early phase code, please refine my code to cope with all other use cases but for now it works).

defmodule PhoenixFailToProcessZipedBodyWeb.Endpoint do
   # ...

  # 1. change your Plug.Parsers options to this, so it accepts
  # our newly created GzipBodyReader that is defined below.
  plug Plug.Parsers,
    parsers: [:urlencoded, :multipart, :json],
    pass: ["*/*"],
    body_reader: {GzipBodyReader, :read_body, []},
    json_decoder: Phoenix.json_library()

    # ...
end

# 2. this is the GzipBodyReader (almost 1:1 from Plug documentation here: 
# https://hexdocs.pm/plug/Plug.Parsers.html#module-custom-body-reader
defmodule GzipBodyReader do
  def read_body(conn, opts) do
    {:ok, body, conn} = Plug.Conn.read_body(conn, opts)
    uncompressed_body = decompress_body(body, "gzip")
    {:ok, uncompressed_body, conn}
  end

  # 3. this part is copied from tesla library decompression part, normally gzip would be your
  # content encoding header so you can also do deflate and pass all others untouched
  # see Teslas code (as linked above in earlier reply) how to handle those cases.
  defp decompress_body(<<31, 139, 8, _::binary>> = body, "gzip"), do: :zlib.gunzip(body)
end

So this seems to work for me.
I also have to point you on a little change that your version of client.ex points to / instead of /create.
(So it actually calls your controller action).

Hope this gets you back on track!
Cheers!

4 Likes

Just FYI I found some related issues on plug and cowboy and wanted to share them:


1 Like

i think you misunderstood this part (or i was not clear enough) these points are the literal steps i use(d) to solve it. I mean you have a ‘test’ client (as you describe) where you first remove the compression from(because its your test client and not the real one) and then make asserts if everything works, in this case json parsing etc) but then and only then you add the extra problem (compression) back in and solve that last part. So it’s more about the steps i would take (and sort of took with my solution and making small changes and asserts along the way). Just saying to help you in the future (as one way, could be better ways) to approach you problems.

@rjk oh … you can implement a body_reader in the plug parser … that’s awsome !

Thank a lot.
Removing Plug.parsers and “do it myself” would have been indeed an horrible solution.

If someone need the same thing, here is a little bit more (beware it’s not safe against case in header keys) :

defmodule GzipBodyReader do
  def read_body(%Plug.Conn{req_headers: req_headers} = conn, opts) do
    {:ok, body, conn} = Plug.Conn.read_body(conn, opts)

    uncompressed_body = decompress_body(body, header_body_encoding(req_headers))

    {:ok, uncompressed_body, conn}
  end

  # 3. this part is copied from tesla library decompression part, normally gzip would be your
  # content encoding header so you can also do deflate and pass all others untouched
  # see Teslas code (as linked above in earlier reply) how to handle those cases.
  defp decompress_body(<<31, 139, 8, _::binary>> = body, :gzip), do: :zlib.gunzip(body)
  defp decompress_body(body, :deflate), do: :zlib.unzip(body)
  defp decompress_body(body, _content_encoding), do: body

  defp header_body_encoding(headers) do
    cond do
      Enum.any?(headers, fn header ->
        header in [{"content-encoding", "gzip"}, {"content-encoding", "x-gzip"}]
      end) ->
        :gzip

      Enum.member?(headers, {"content-encoding", "deflate"}) ->
        :deflate

      true ->
        :none
    end
  end
end
1 Like

I had to implement gzip request handling and found this thread useful. I put together a small library based on the discussion I found here. Right now, the lib only handles content-encoding: gzip, but it can easily be extended to handle more content-encodings.

:zlib.gunzip/1 is great if you know you can trust the input, but if your endpoint is exposed to the internet, you could be vulnerable to a zip bomb attack, so instead I used :zlib.safeInflate/2. I hope someone finds it useful.

3 Likes