Upgraded to Plug 1.9.0 from 1.8.3 - invalid UTF-8 on urlencoded params

Hi,

I upgraded to Plug 1.9.0 from 1.8.3 and now my tests are failing with the error below

(Plug.Conn.InvalidQueryError) invalid UTF-8 on urlencoded params, got byte 254

In the Plug changelog it says “Add option to skip utf8 validation”

Does anyone know how to skip utf8 validation on certain routes?

Nick

The code point 254 is valid UTF-8 but its not valid in urlencoding. Probably a bit pedantic given it doesn’t solve your problem. But there are potentially other issues if you are receiving invalid URLs.

I’m sure someone more knowledgable than I will answer the question you actually asked :slight_smile:

The byte 254 is not valid UTF-8. It is declared as invalid by the spec.

The Unicode Codepoint 254 is indeed valid, but would be 2 bytes in UTF-8: <<0xC9, 0x94>>

1 Like

If your tests are failing with that error, you might need to show us your tests, then we might be able to fix them with you.

As @kip already said, if you currently rely on receiving malformed data, then you should probably fix your receiver and tell your providers to send in the data correctly specified/escaped.

The tests are checking real world requests from an external system, they are SMS replies and are UCS2 encoded and they can contain control characters.

Below is what we do currently to decode the message, if its multipart its more complicated

query_string = "message=%00S%00m%00i%00l%00e%00%20%26%3A%FE%0F%00%20%00g%00r%00i%00n%00%20%D8%3D%DE%01%00%20%00w\
%00i%00n%00k%00%20%D8%3D%DE%09&mobileno=0&recipient=0&type=mo&charset=UCS2&dcs=08"

r = URI.decode_query(query_string)
:unicode.characters_to_binary(r["message"], {:utf16, :big})

I was able to fix it by adding the below to endpoint.ex above the parsers

  defp stop_parsers_for_hsl_requests(conn, _) do
    case conn.path_info do
      ["api", "v1", "webhooks", "hsl"] ->
        %{conn | params: %{}, query_params: URI.decode_query(conn.query_string), body_params: %{}}

      _ ->
        conn
    end
  end

  plug :stop_parsers_for_hsl_requests

This external system sends invalid requests. Even though the byte 254 might be valid UCS2, it is not valid in a query string and has to be escaped as %FE.

Please file a bug upstream, while they fix it, you can use your temporary workaround.

PS, the example input you have shown does not even contain that byte.

1 Like