"HTTP/1.1 431 Request Header Fields Too Large" with many cookies only in Mix.env==:prod

Phoenix Forum Questions / Help
, ,
1

I’m having some weird issue when people make requests with many cookies:

2020-09-17 19:12:50.285 [error] Cowboy returned 431 because it was unable to parse the request headers.This may happen because there are no headers, or there are too many headers
or the header name or value are too large (such as a large cookie).

More specific reason is:

    :"A header value is larger than configuration allows. (RFC7230 3.2.5, RFC6585 5)"

You can customize those limits when configuring your http/https server. The configuration option and default values are shown below:

    protocol_options: [
      max_header_name_length: 64,
      max_header_value_length: 4096,
      max_headers: 100
    ]

so I added this bit to my config/config.exs:

config :horse_racing_nation_web, HorseRacingNationWeb.News.Endpoint,
  http: [protocol_options: [max_header_value_length: 8192]],
  ...

…which fixes it in development, but when I build a release I still get the 431.

I can see the option makes it through to the sys.config in the release:

   {'Elixir.HorseRacingNationWeb.News.Endpoint',
       [{http,
            [{protocol_options,
                 [{max_header_value_length,8192}]}]},

Any ideas why it works in development but not production?

2

:wave:

What do get you from

:ranch.get_protocol_options(HorseRacingNationWeb.News.Endpoint.HTTP)
# or :ranch.get_protocol_options(HorseRacingNationWeb.News.Endpoint.HTTPS)

? Does the new value for max_header_value_length show up there?

3

Probably because your http config in prod is overwriting the config since it has :inet6 as the first element thus it’s not a keyword list.

1 Like
4

FWIW, many user agents are still going to have the limit from the RFC regardless of what the server thinks. Consider storing that data elsewhere to avoid compatibility hassles.

5

I just checked, and we don’t have the :inet6 in our config. The app also doesn’t have https configured, because it’s terminated in the load balancer. We’re also not using prod.secret.exs.

6

Here’s what I got:

iex(dev@127.0.0.1)1> :ranch.get_protocol_options(HorseRacingNationWeb.News.Endpoint.HTTP)
%{
  env: %{
    dispatch: [
      {:_, [],
       [
         {:_, [], Phoenix.Endpoint.Cowboy2Handler,
          {HorseRacingNationWeb.News.Endpoint, []}}
       ]}
    ]
  },
  stream_handlers: [Plug.Cowboy.Stream]
}

It looks like the config is not making it through!

7

I made a mistake @chulkilee, I found the :inet6 inside the endpoint rather than the config:

  def init(_key, config) do
    if config[:load_from_system_env] do
      port =
        System.get_env("NEWS_PORT") ||
          raise "expected the NEWS_PORT environment variable to be set"

      {:ok, Keyword.put(config, :http, [:inet6, port: port])}
    else
      {:ok, config}
    end
  end

Making the change in your first link now. It’ll take me a bit to verify.

8

I got the fix working with this change:

         System.get_env("NEWS_PORT") ||
           raise "expected the NEWS_PORT environment variable to be set"
 
-      {:ok, Keyword.put(config, :http, [:inet6, port: port])}
+      {:ok, merge(config, http: [port: port, transport_options: [socket_opts: [:inet6]]])}
     else
       {:ok, config}
     end
   end
+
+  @doc """
+  Take 2 keyword lists and merge them recursively.
+  Used to merge configuration values into defaults.
+  """
+  defp merge(a, b), do: Keyword.merge(a, b, &merger/3)
+
+  defp merger(_k, v1, v2) do
+    if Keyword.keyword?(v1) and Keyword.keyword?(v2) do
+      Keyword.merge(v1, v2, &merger/3)
+    else
+      v2
+    end
+  end
 end