You don’t actually have one backslash in the original string. See:

iex(9)> body = %{password: "my\Password"}
%{password: "myPassword"}

Notice how the output is just "myPassword". The backslash is gone there, it doesn’t have anything to do with Poison’s JSON encoding. \ is itself the escape character in a string. If you want one in there literally, you need to escape it:

iex(1)> IO.puts("my\Password")
myPassword
:ok
iex(2)> IO.puts("my\\Password")
my\Password
:ok

Notably, if you’re reading in external text that has a slash, this will not be lost:

iex(3)> string = IO.gets("Password: ")
Password: my\Password
"my\\Password\n"
iex(4)> IO.puts string
my\Password

It is correctly encoded with double backslash and only one backslash is sent in request.
Multiple backslashes are used because both Elixir and JSON use them for escaping special characters.
This is just visual representation in terminal.

iex(1)> body = %{password: "my\\Password"}
%{password: "my\\Password"}
iex(2)> Poison.encode!(body)              
"{\"password\":\"my\\\\Password\"}"
iex(3)> Poison.encode!(body) |> IO.puts   
{"password":"my\\Password"}
:ok

Last result is a JSON that will be sent to the server.

body = %{password: ~S(my\Password)} and you don’t need to escape the backslash. Sigils can be wrapped by many different characters so unless you use all of them you should find one, which isn’t problematic: Syntax reference — Elixir v1.20.2