Torrent client sending non UTF-8 character in urlencoded params, got byte 233

Phoenix Forum Questions / Help
, ,
1

So, just to preface this post, I’m attempting to build an HTTP torrent tracker in Elixir. I have also been able to successfully decode and reencode multiple torrent files with the bencode and bento packages.

The issue comes about when I attempt to bring phoenix into the mix with the “invalid” urlencoded params, and saying it “got byte 233”. As of right now I have identified the info_hash key in the form of an urlencoded 20-byte SHA1 hash that has been bencoded. ref: Bit Torrent Specification - Tracker Request Parameters

I have already attempted to disable UTF-8 validation in Plug.Parsers with the validate_utf8 option, but that seemingly had no effect. I have also attempted to outright disable Plug.Parsers all together with the same result.

[debug] ** (Plug.Conn.InvalidQueryError) invalid UTF-8 on urlencoded params, got byte 233
    (plug 1.13.6) lib/plug/conn/utils.ex:292: Plug.Conn.Utils.do_validate_utf8!/3
    (plug 1.13.6) lib/plug/conn/query.ex:116: Plug.Conn.Query.decode_www_form/3
    (plug 1.13.6) lib/plug/conn/query.ex:97: Plug.Conn.Query.decode_www_pair/4
    (elixir 1.14.0) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (plug 1.13.6) lib/plug/conn.ex:986: Plug.Conn.fetch_query_params/2
    (phoenix_live_dashboard 0.6.5) lib/phoenix/live_dashboard/request_logger.ex:35: Phoenix.LiveDashboard.RequestLogger.verify_from_param_key/2
    (phoenix_live_dashboard 0.6.5) lib/phoenix/live_dashboard/request_logger.ex:27: Phoenix.LiveDashboard.RequestLogger.call/2
    (ex_torrent 0.1.0) lib/ex_torrent_web/endpoint.ex:1: ExTorrentWeb.Endpoint.plug_builder_call/2
    (ex_torrent 0.1.0) lib/plug/debugger.ex:136: ExTorrentWeb.Endpoint."call (overridable 3)"/2
    (ex_torrent 0.1.0) lib/ex_torrent_web/endpoint.ex:1: ExTorrentWeb.Endpoint.call/2
    (phoenix 1.6.12) lib/phoenix/endpoint/cowboy2_handler.ex:54: Phoenix.Endpoint.Cowboy2Handler.init/4
    (cowboy 2.9.0) /home/scott/Desktop/ex_torrent/deps/cowboy/src/cowboy_handler.erl:37: :cowboy_handler.execute/2
    (cowboy 2.9.0) /home/scott/Desktop/ex_torrent/deps/cowboy/src/cowboy_stream_h.erl:306: :cowboy_stream_h.execute/3
    (cowboy 2.9.0) /home/scott/Desktop/ex_torrent/deps/cowboy/src/cowboy_stream_h.erl:295: :cowboy_stream_h.request_process/3
    (stdlib 4.0.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
2

Help us understand: you’re making a web app (HTTP server/listener) and it gets called by other services with invalid URL-encoded data?

3

Torrent trackers announce URLs get called directly by torrent clients, Transmission BT and µTorrent are examples of this. Basically, all this service will do is keep track (as the name implies) of the torrent hash and reply back with the other clients also listening in on the same hash. After the response is sent back, the clients will talk to each other for P2P file transfer.

I have an example URI with the key that is causing the issue with an example of what is causing the InvalidQueryError.

localhost:4000/announce?info_hash=%13%E9%91%E8%998%C8%E2k%1A%10%C2%BE%154it%E3%5E%CF

Here is a very simple golang example of a torrent tracker: https://github.com/meehow/privtracker

4

fetch_query_params has an option to disable utf8 validation, which you likely want to set, given this seems to try to supply raw bytes and not utf8 characters.

5

Do you happen to have a link to documentation for this?
I was looking through the Plug docs last night thinking that might have a possible solution but didn’t find anything that ended up helping.

6

Plug.Conn — Plug v1.13.6 as well as Plug.Parsers — Plug v1.13.6

7

Thank you for the links!
I have tried the Plug.Parsers way as mentioned in the initial post, and haven’t had any success by disabling utf8 validation in the endpoint or outright removing the Plug.Parsers from the endpoint.

I will try to see if I can find out how exactly I can use fetch_query_params since I’ve never had to use it before, I will post an update with results!

8

Many torrent clients will send raw binary as the hash in the url. Can you skip the parsing from plug and just get the raw URL binary so you can decode it yourself ?

9

That is what I am attempting to do, but seemingly even with the Plug.Parsers plug commented out in the endpoint.ex file, it will still try to validate utf8 with the same result as the original logs.

Here is what the router currently looks like, and I tried the :fetch_query_params option that was suggested by LostKobrakai, with the same result.

  pipeline :api do
    plug :fetch_query_params, validate_utf8: false
    plug :accepts, ["*/*"]
  end

  scope "/", ExTorrentWeb do
    pipe_through :api

    get "/announce", PageController, :announce
  end

And here is the controller, nothing that should cause any issues in here, basically a blank slate project.

defmodule ExTorrentWeb.PageController do
  use ExTorrentWeb, :controller

  def announce(conn, params) do
    IO.inspect(params)

    conn
    |> put_status(200)
    |> json(%{success: true})
    |> halt()
  end
end
10

Looking at the error again it seems the issue is the live dashboard request logger trying to fetch query params (to enable itself if the necessary query param is available).

11

Wow, what a strange bug, yeah disabling the Phoenix.LiveDashboard.RequestLogger in the endpoint fixed the issue.
I’m going to submit an issue upstream on the Live Dashboard repo.