cgrimmett

cgrimmett

Creating a BEP3 compatible percent encoder (URI encoder)

I would like to announce torrents to my opentracker using an Elixir program. I have a test qbittorrent and opentracker running and I’ve identified how qbittorrent sends it’s announces to opentracker.

METHOD: GET
URL
http://localhost:8000/announce?info_hash=%ac%c3%b2%e43%d7%c7GZ%bbYA%b5h%1c%b7%a1%ea%26%e2&peer_id=-qB5020-r3FSX0qNU6Oo&port=4993&uploaded=0&downloaded=0&left=0&corrupt=0&key=F7F879A3&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0
HEADERS
Accept-Encoding:
gzip
Connection:
close
Host:
localhost:8000
User-Agent:
qBittorrent/5.0.2

I would like to build this same GET request into my own code. My confusion comes from the percent encoded info_hash. Neither URI.encode/2 nor URI.encode_www_form/1 encode the URI the same way that qbittorrent does it.

Let me show you what I’ve tried so far.


# We start with the `Info hash v1` of the torrent, as copied from qBittorrent.
info_hash_v1 = "acc3b2e433d7c7475abb5941b5681cb7a1ea26e2"

# Next we decode the hexadecimal representation into binary.
binary = Base.decode16!(info_hash_v1, case: :lower)

I’m not sure what comes next. Digging through Elixir URI source code, I can see the method of percent encoding that creates a binary in almost the right format.

https://github.com/elixir-lang/elixir/blob/78f63d08313677a680868685701ae79a2459dcc1/lib/elixir/lib/uri.ex#L428C7-L428C13

The problem here is that the hex/1 function only outputs 16 possible values, A-F, 0-9. There is no lowercase! The way qbittorrent is doing their percent encoding, they have lowercase too-- a-f, A-F, 0-9.

I read qbittorrent source code to see how they’re percent encoding, but I couldn’t find the code that does that. I found some info_hash references though. I can barely read C++, but I think the url encoding might be abstracted away in a request library.

I also looked through transmission-qt code. That code is greek to me, but I was able to find their percent encoder implementation.

https://github.com/transmission/transmission/blob/87bcf1a1d5bc60070315335a85fb8526843b28ac/libtransmission/web-utils.h#L100

Seems similar to what I read in Elixir URI source code related to unreserved and unescaped characters. RFC 3986 - Uniform Resource Identifier (URI): Generic Syntax

I bounced some ideas off of ChatGPT, borrowed code from Elixir URI, and I got a bep3_encode/1 function put together.


  @doc """
  Encodes `string` as BEP3's weird URL encoded string.

  ## Example

      iex> bep3_encode("a88fda5954e89178c372716a6a78b8180ed4dad3")
      "%A8%8F%DAYT%E8%91x%C3rqjjx%B8%18%0E%D4%DA%D3"

  """
  @spec bep3_encode(binary) :: binary
  def bep3_encode(string) when is_binary(string) do
    string = Base.decode16!(string, case: :lower)
    URI.encode(string, &URI.char_unreserved?/1)
  end

Given an Info hash v1 input of acc3b2e433d7c7475abb5941b5681cb7a1ea26e2, the expected output should be as follows.

%ac%c3%b2%e43%d7%c7GZ%bbYA%b5h%1c%b7%a1%ea%26%e2

However, I haven’t figured out how I preserve the case on the various letters. I assume I need to have exactly the same case-sensitive output as qBittorrent because an ASCII A is not the same as a.

It seems like the way qbittorrent does it, and I apologize for not having the language to communicate this.. hex values that can be displayed as ASCII are output as ASCII. Otherwise, the hex representation is displayed.

I wrote this out to make a visual comparison of expected input and output values, because this is the only way I could think of understanding what is happening during the encoding.

 a8   8f   da   59   54   e8   91   78   c3   72   71   6a   6a   78   b8   18   0e   d4   da   d3
%A8  %8F  %DA   Y    T   %E8  %91   x   %C3   r    q    j    j    x   %B8  %18  %0E  %D4  %DA  %D3

On the second line (expected output) See the ASCII Y? That matches up with Hex 54. The three values before that were all not able to be displayed as ASCII, so the hex value was used instead.

Later on, we can see lowercase x, r, q, j, j, x. From what I can tell, Elixir’s built-in URI.encode/1 can’t do this, because like I mentioned earlier, that can only output A-F,0-9. No lowercase!

By the way, https://www.asciitohex.com/ has been very helpful.

Anyway, I am very confused. I’m going to rest now and pick this up in the morning.

Marked As Solved

cgrimmett

cgrimmett

Eureka!

defmodule App.BittorrentUrlEncoder do
  @moduledoc """
  URL encoding for Bittorrent Info hash v1. Designed to be compatible with qBittorrent's percent encoding.
  """

  import Bitwise

  @doc """
  Encodes `string` as a Bittorrent-flavored percent-encoded string.

  ## Example

      iex> encode("a88fda5954e89178c372716a6a78b8180ed4dad3")
      "%a8%8f%daYT%e8%91x%c3rqjjx%b8%18%0e%d4%da%d3"

  """
  @spec encode(binary()) :: binary()
  def encode(hex_string) when is_binary(hex_string) do
    hex_string
    |> Base.decode16!(case: :lower) # Decode from hex to raw bytes
    |> encode_bytes()
  end

  defp encode_bytes(<<>>), do: ""

  defp encode_bytes(<<byte, rest::binary>>) do
    percent_encode(byte) <> encode_bytes(rest)
  end

  defp percent_encode(byte) when byte in ?0..?9 or byte in ?a..?z or byte in ?A..?Z or byte in ~c"~_-.!" do
    <<byte>>
  end

  defp percent_encode(byte) do
    "%" <> <<hex(bsr(byte, 4)), hex(band(byte, 15))>>
  end

  defp hex(n) when n <= 9, do: n + ?0
  defp hex(n), do: n + ?a - 10
end

The secret sauce was changing ?A to ?a in the hex/1 function

-defp hex(n), do: n + ?A - 10
+defp hex(n), do: n + ?a - 10

Also Liked

al2o3cr

al2o3cr

This is not true, as trying it will demonstrate:

iex(1)> s = <<0xA8, 0x59, 0x72>>
<<168, 89, 114>>

iex(2)> URI.encode(s)
"%A8Yr"

Where Next?

Popular in Questions Top

New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
mgjohns61585
Could someone help me? I’m making my first elixir program, number guessing game. I can’t figure out how to convert the user’s guess from ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
vac
Hi, I’m quite new in Elixir and I’m trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and I...
New
earth10
Hi, I’m just starting to build a side-project with Elixir and Phoenix and doing some basic test with Elixir alone. What strikes me is th...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
beno
I will often find my self writing things similar to: case some_value do nil -&gt; something() "" -&gt; something() _ -&gt; somethi...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: The documentation above suggests that while ...
New

Other popular topics Top

johnnyicon
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
Lily
In templates/appointment/index.html.eex: &lt;%= for appointment &lt;- @appointments do %&gt; &lt;tr&gt; &lt;td&gt;&lt;%= appoi...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31142 143
New
boundedvariable
I am going through the kafka architecture. All the features what the kafka is providing are already in Erlang. I would like hear your opi...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

We're in Beta

About us Mission Statement