Content-encoding gzip auto deflate/gunzip body on incoming requests

isnt this something in the lines of:

config :foo, FooWeb.Endpoint,
  http: [
    compress: true,
    protocol_options: [max_keepalive: 5_000_000]
  ],

(The compress: true part?)

Also it looks like compress: true is to compress response from server, not about incomming zipped body.

@rjk Tried it, same behaviour, here is what I have in my Endpoint conf:

iex(3)> Application.get_env(:my_app, MyappWeb.Endpoint)

[
  url: [host: "localhost"],
  secret_key_base: "******",
  render_errors: [view: MyappWeb.ErrorView, accepts: ["json"], layout: false],
  pubsub_server: Myapp.PubSub,
  live_view: [signing_salt: "****"],
  http: [port: 4000, compress: true],
  debug_errors: true,
  code_reloader: true,
  check_origin: false,
  watchers: []
]

Is the request containing a file that needs to be compressed, or you’re after the whole HTTP request being gzipped when being sent from the client?

you’re completely right, I responded too fast.

Context around this is that there are still open tickets/request in cowboy(http server) and Plug about this, the summary is that it’s a bit tricky because of zip bombs (AFAICS). But this should not stop you for your own API.

How I would go about this for now:

• Remove compression from the client calls (comment that tesla plug in the client)
• I would directly point your HTTP client calls to your controller (removing the JSON parser pipeline part)
  so now you get the client request ‘raw’ into your controller
• now JSON decode it yourself inside that controller/action
• if that works now add compression back in (uncomment that tesla compression plug again in that client)
• now i expect the same error again as before about ‘cannot parse json’ (but now you can fix it completely inside that controller action itself)
• add in the gunzipping of the body in the same way Tesla does it, see here (tesla/lib/tesla/middleware/compression.ex at v1.3.3 · elixir-tesla/tesla · GitHub) so you’ll have something along the lines of :zlib.gunzip body of the request inside your controller action.

now you have it working and can refactor it with; create a Plug (you can add a plug folder in the same folder as where controllers/templates/views folders are located) and now in the call action from your plug you only add back in the gunzipping part. Now you can go back to your old code where you had it going through that API pipeline with json plug but put that gunzip plug before it.

Seems a bit of work but will be worth it in the end

ps. please be aware that if you do larger multipart/‘streaming’ posts from the client you have to handle it a bit differently and this could surprise you.

ps2. If it’s now working out for you, try to publish your code into the open (e.g. github) as a minimal example because then we can help you way better and respond with something which is directly usable in form of code commits / verified to work

@dimitarvp the request I’m receiving on my api contain a ziped json as a body, with following header:

{“content-encoding”, “gzip”},
{“content-type”, “application/json”}

@rjk: of course I can’t controller the client side, like you say the solution would be to remove the Plug.Parsers (JSON decode) from Endpoint.ex and handeling it myself.
I would be ideal to create a new plug that I would be before Plug.Parsers that gunzip if needed according to conn header. I started doing that but I have trouble to set to body back in the conn, it looks like it’s not to correct way of doing it.

I’ve created GitHub - xward/phoenix_fail_to_process_ziped_body · GitHub for you, will be very easy to reproduce (cf readme.md)

Thanks !

Ok got it working, the JSON decoding part is ‘lower’ in the plug pipeline than we thought, you can see it in your endpoint in the parsing bit. That’s also the place where I got it working (early phase code, please refine my code to cope with all other use cases but for now it works).

defmodule PhoenixFailToProcessZipedBodyWeb.Endpoint do
   # ...

  # 1. change your Plug.Parsers options to this, so it accepts
  # our newly created GzipBodyReader that is defined below.
  plug Plug.Parsers,
    parsers: [:urlencoded, :multipart, :json],
    pass: ["*/*"],
    body_reader: {GzipBodyReader, :read_body, []},
    json_decoder: Phoenix.json_library()

    # ...
end

# 2. this is the GzipBodyReader (almost 1:1 from Plug documentation here: 
# https://hexdocs.pm/plug/Plug.Parsers.html#module-custom-body-reader
defmodule GzipBodyReader do
  def read_body(conn, opts) do
    {:ok, body, conn} = Plug.Conn.read_body(conn, opts)
    uncompressed_body = decompress_body(body, "gzip")
    {:ok, uncompressed_body, conn}
  end

  # 3. this part is copied from tesla library decompression part, normally gzip would be your
  # content encoding header so you can also do deflate and pass all others untouched
  # see Teslas code (as linked above in earlier reply) how to handle those cases.
  defp decompress_body(<<31, 139, 8, _::binary>> = body, "gzip"), do: :zlib.gunzip(body)
end

So this seems to work for me.
I also have to point you on a little change that your version of client.ex points to / instead of /create.
(So it actually calls your controller action).

Hope this gets you back on track!
Cheers!

Just FYI I found some related issues on plug and cowboy and wanted to share them:

https://github.com/elixir-plug/plug/issues/886
https://github.com/ninenines/cowboy/issues/946

i think you misunderstood this part (or i was not clear enough) these points are the literal steps i use(d) to solve it. I mean you have a ‘test’ client (as you describe) where you first remove the compression from(because its your test client and not the real one) and then make asserts if everything works, in this case json parsing etc) but then and only then you add the extra problem (compression) back in and solve that last part. So it’s more about the steps i would take (and sort of took with my solution and making small changes and asserts along the way). Just saying to help you in the future (as one way, could be better ways) to approach you problems.

@rjk oh .. you can implement a body_reader in the plug parser … that’s awsome !

Thank a lot.
Removing Plug.parsers and “do it myself” would have been indeed an horrible solution.

If someone need the same thing, here is a little bit more (beware it’s not safe against case in header keys) :

defmodule GzipBodyReader do
  def read_body(%Plug.Conn{req_headers: req_headers} = conn, opts) do
    {:ok, body, conn} = Plug.Conn.read_body(conn, opts)

    uncompressed_body = decompress_body(body, header_body_encoding(req_headers))

    {:ok, uncompressed_body, conn}
  end

  # 3. this part is copied from tesla library decompression part, normally gzip would be your
  # content encoding header so you can also do deflate and pass all others untouched
  # see Teslas code (as linked above in earlier reply) how to handle those cases.
  defp decompress_body(<<31, 139, 8, _::binary>> = body, :gzip), do: :zlib.gunzip(body)
  defp decompress_body(body, :deflate), do: :zlib.unzip(body)
  defp decompress_body(body, _content_encoding), do: body

  defp header_body_encoding(headers) do
    cond do
      Enum.any?(headers, fn header ->
        header in [{"content-encoding", "gzip"}, {"content-encoding", "x-gzip"}]
      end) ->
        :gzip

      Enum.member?(headers, {"content-encoding", "deflate"}) ->
        :deflate

      true ->
        :none
    end
  end
end