xward

Content-encoding gzip auto deflate/gunzip body on incoming requests

I would like to be avoid to receive content-encoding gzip and have transparent gzip/deflate when content reach my controller.

Using
phoenix 1.5.0
elixir 1.10.2
erlang 22.3
jason 1.0
plug_cowboy 2.0
tesla 1.3

Here is some code to show you what I see:

Client side:

defmodule Client do
  use Tesla

  plug(Tesla.Middleware.JSON)
  plug(Tesla.Middleware.Compression)

  def post, do: post("localhost:4000", %{hello: :i_m_fine})
end

Server side:

defmodule Router do
  use MyAppWeb, :router

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/test_notification" do
    pipe_through :api
    post("/", MyAppWeb.TestController, :test)
  end
end

defmodule MyAppWeb.TestController do
  @moduledoc false
  use MyAppWeb, :controller

  def test(conn, _args) do
    conn
    |> send_resp(200, "yeah me too !")
    |> halt()
  end
end

Doing a request:

# in my conn headers
req_headers: [
  {"connection", "keep-alive"},
  {"content-encoding", "gzip"},
  {"content-length", "123"},
  {"content-type", "application/json"},
  {"host", "localhost:4000"}
]

# fail before reaching my controller, fail in Plug.Parser that expect to Json decode

** (Plug.Parsers.ParseError) malformed request, a Jason.DecodeError exception was raised with message "unexpected byte at position 0: 0x1F"
    (plug 1.10.1) lib/plug/parsers/json.ex:88: Plug.Parsers.JSON.decode/2
    (plug 1.10.1) lib/plug/parsers.ex:313: Plug.Parsers.reduce/8
    (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.plug_builder_call/2
    (my_app 0.1.0) lib/plug/debugger.ex:132: MyAppWeb.Endpoint."call (overridable 3)"/2
    (my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.call/2
    (phoenix 1.5.3) lib/phoenix/endpoint/cowboy2_handler.ex:65: Phoenix.Endpoint.Cowboy2Handler.init/4

I could add my own little plug in Enpoint.ex that does the feature, but I feel like I miss an obvious option to configure my phoenix endpoint.

Have a nice day everyone !

13 comments

/phoenix #troubleshooting #encoding

13 4819 13

2025-03-09 22:58:04 UTC

Marked As Solved

rjk

Ok got it working, the JSON decoding part is ‘lower’ in the plug pipeline than we thought, you can see it in your endpoint in the parsing bit. That’s also the place where I got it working (early phase code, please refine my code to cope with all other use cases but for now it works).

defmodule PhoenixFailToProcessZipedBodyWeb.Endpoint do
   # ...

  # 1. change your Plug.Parsers options to this, so it accepts
  # our newly created GzipBodyReader that is defined below.
  plug Plug.Parsers,
    parsers: [:urlencoded, :multipart, :json],
    pass: ["*/*"],
    body_reader: {GzipBodyReader, :read_body, []},
    json_decoder: Phoenix.json_library()

    # ...
end

# 2. this is the GzipBodyReader (almost 1:1 from Plug documentation here: 
# https://hexdocs.pm/plug/Plug.Parsers.html#module-custom-body-reader
defmodule GzipBodyReader do
  def read_body(conn, opts) do
    {:ok, body, conn} = Plug.Conn.read_body(conn, opts)
    uncompressed_body = decompress_body(body, "gzip")
    {:ok, uncompressed_body, conn}
  end

  # 3. this part is copied from tesla library decompression part, normally gzip would be your
  # content encoding header so you can also do deflate and pass all others untouched
  # see Teslas code (as linked above in earlier reply) how to handle those cases.
  defp decompress_body(<<31, 139, 8, _::binary>> = body, "gzip"), do: :zlib.gunzip(body)
end

So this seems to work for me.
I also have to point you on a little change that your version of client.ex points to / instead of /create.
(So it actually calls your controller action).

Hope this gets you back on track!
Cheers!

Post #8

Also Liked

sneako

I had to implement gzip request handling and found this thread useful. I put together a small library based on the discussion I found here. Right now, the lib only handles content-encoding: gzip, but it can easily be extended to handle more content-encodings.

:zlib.gunzip/1 is great if you know you can trust the input, but if your endpoint is exposed to the internet, you could be vulnerable to a zip bomb attack, so instead I used :zlib.safeInflate/2. I hope someone finds it useful.

https://github.com/sneako/plug_compressed_body_reader

Post #12