I would like to be avoid to receive content-encoding gzip and have transparent gzip/deflate when content reach my controller.
Using
phoenix 1.5.0
elixir 1.10.2
erlang 22.3
jason 1.0
plug_cowboy 2.0
tesla 1.3
Here is some code to show you what I see:
Client side:
defmodule Client do
use Tesla
plug(Tesla.Middleware.JSON)
plug(Tesla.Middleware.Compression)
def post, do: post("localhost:4000", %{hello: :i_m_fine})
end
Server side:
defmodule Router do
use MyAppWeb, :router
pipeline :api do
plug :accepts, ["json"]
end
scope "/test_notification" do
pipe_through :api
post("/", MyAppWeb.TestController, :test)
end
end
defmodule MyAppWeb.TestController do
@moduledoc false
use MyAppWeb, :controller
def test(conn, _args) do
conn
|> send_resp(200, "yeah me too !")
|> halt()
end
end
Doing a request:
# in my conn headers
req_headers: [
{"connection", "keep-alive"},
{"content-encoding", "gzip"},
{"content-length", "123"},
{"content-type", "application/json"},
{"host", "localhost:4000"}
]
# fail before reaching my controller, fail in Plug.Parser that expect to Json decode
** (Plug.Parsers.ParseError) malformed request, a Jason.DecodeError exception was raised with message "unexpected byte at position 0: 0x1F"
(plug 1.10.1) lib/plug/parsers/json.ex:88: Plug.Parsers.JSON.decode/2
(plug 1.10.1) lib/plug/parsers.ex:313: Plug.Parsers.reduce/8
(my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.plug_builder_call/2
(my_app 0.1.0) lib/plug/debugger.ex:132: MyAppWeb.Endpoint."call (overridable 3)"/2
(my_app 0.1.0) lib/my_app_web/endpoint.ex:1: MyAppWeb.Endpoint.call/2
(phoenix 1.5.3) lib/phoenix/endpoint/cowboy2_handler.ex:65: Phoenix.Endpoint.Cowboy2Handler.init/4
I could add my own little plug in Enpoint.ex that does the feature, but I feel like I miss an obvious option to configure my phoenix endpoint.
Context around this is that there are still open tickets/request in cowboy(http server) and Plug about this, the summary is that itâs a bit tricky because of zip bombs (AFAICS). But this should not stop you for your own API.
How I would go about this for now:
Remove compression from the client calls (comment that tesla plug in the client)
I would directly point your HTTP client calls to your controller (removing the JSON parser pipeline part)
so now you get the client request ârawâ into your controller
now JSON decode it yourself inside that controller/action
if that works now add compression back in (uncomment that tesla compression plug again in that client)
now i expect the same error again as before about âcannot parse jsonâ (but now you can fix it completely inside that controller action itself)
now you have it working and can refactor it with; create a Plug (you can add a plug folder in the same folder as where controllers/templates/views folders are located) and now in the call action from your plug you only add back in the gunzipping part. Now you can go back to your old code where you had it going through that API pipeline with json plug but put that gunzip plug before it.
Seems a bit of work but will be worth it in the end
ps. please be aware that if you do larger multipart/âstreamingâ posts from the client you have to handle it a bit differently and this could surprise you.
ps2. If itâs now working out for you, try to publish your code into the open (e.g. github) as a minimal example because then we can help you way better and respond with something which is directly usable in form of code commits / verified to work
@rjk: of course I canât controller the client side, like you say the solution would be to remove the Plug.Parsers (JSON decode) from Endpoint.ex and handeling it myself.
I would be ideal to create a new plug that I would be before Plug.Parsers that gunzip if needed according to conn header. I started doing that but I have trouble to set to body back in the conn, it looks like itâs not to correct way of doing it.
Ok got it working, the JSON decoding part is âlowerâ in the plug pipeline than we thought, you can see it in your endpoint in the parsing bit. Thatâs also the place where I got it working (early phase code, please refine my code to cope with all other use cases but for now it works).
defmodule PhoenixFailToProcessZipedBodyWeb.Endpoint do
# ...
# 1. change your Plug.Parsers options to this, so it accepts
# our newly created GzipBodyReader that is defined below.
plug Plug.Parsers,
parsers: [:urlencoded, :multipart, :json],
pass: ["*/*"],
body_reader: {GzipBodyReader, :read_body, []},
json_decoder: Phoenix.json_library()
# ...
end
# 2. this is the GzipBodyReader (almost 1:1 from Plug documentation here:
# https://hexdocs.pm/plug/Plug.Parsers.html#module-custom-body-reader
defmodule GzipBodyReader do
def read_body(conn, opts) do
{:ok, body, conn} = Plug.Conn.read_body(conn, opts)
uncompressed_body = decompress_body(body, "gzip")
{:ok, uncompressed_body, conn}
end
# 3. this part is copied from tesla library decompression part, normally gzip would be your
# content encoding header so you can also do deflate and pass all others untouched
# see Teslas code (as linked above in earlier reply) how to handle those cases.
defp decompress_body(<<31, 139, 8, _::binary>> = body, "gzip"), do: :zlib.gunzip(body)
end
So this seems to work for me.
I also have to point you on a little change that your version of client.ex points to / instead of /create.
(So it actually calls your controller action).
i think you misunderstood this part (or i was not clear enough) these points are the literal steps i use(d) to solve it. I mean you have a âtestâ client (as you describe) where you first remove the compression from(because its your test client and not the real one) and then make asserts if everything works, in this case json parsing etc) but then and only then you add the extra problem (compression) back in and solve that last part. So itâs more about the steps i would take (and sort of took with my solution and making small changes and asserts along the way). Just saying to help you in the future (as one way, could be better ways) to approach you problems.
@rjk oh ⊠you can implement a body_reader in the plug parser ⊠thatâs awsome !
Thank a lot.
Removing Plug.parsers and âdo it myselfâ would have been indeed an horrible solution.
If someone need the same thing, here is a little bit more (beware itâs not safe against case in header keys) :
defmodule GzipBodyReader do
def read_body(%Plug.Conn{req_headers: req_headers} = conn, opts) do
{:ok, body, conn} = Plug.Conn.read_body(conn, opts)
uncompressed_body = decompress_body(body, header_body_encoding(req_headers))
{:ok, uncompressed_body, conn}
end
# 3. this part is copied from tesla library decompression part, normally gzip would be your
# content encoding header so you can also do deflate and pass all others untouched
# see Teslas code (as linked above in earlier reply) how to handle those cases.
defp decompress_body(<<31, 139, 8, _::binary>> = body, :gzip), do: :zlib.gunzip(body)
defp decompress_body(body, :deflate), do: :zlib.unzip(body)
defp decompress_body(body, _content_encoding), do: body
defp header_body_encoding(headers) do
cond do
Enum.any?(headers, fn header ->
header in [{"content-encoding", "gzip"}, {"content-encoding", "x-gzip"}]
end) ->
:gzip
Enum.member?(headers, {"content-encoding", "deflate"}) ->
:deflate
true ->
:none
end
end
end
I had to implement gzip request handling and found this thread useful. I put together a small library based on the discussion I found here. Right now, the lib only handles content-encoding: gzip, but it can easily be extended to handle more content-encodings.
:zlib.gunzip/1 is great if you know you can trust the input, but if your endpoint is exposed to the internet, you could be vulnerable to a zip bomb attack, so instead I used :zlib.safeInflate/2. I hope someone finds it useful.
