Httpoison get request with basic authorization -- "HTTP request failed: bad_request"

matthias_toepp · August 6, 2023, 6:39pm

I’m trying to get html from an old machine.

I’m able to successfully get a response on the command line with this (and I have a go program working as well):

curl 200.200.200.200/aaa/001 --user "user:password"

I get the error: “HTTP request failed: bad_request” when trying to do that with an Elixir script:

HTTPoison.start()

username = "user"
password = "password"

credentials_encoded = Base.encode64("#{username}:#{password}")

headers = [{"Authorization", "Basic #{credentials_encoded}"}]

case HTTPoison.get("200.200.200.200/aaa/001", headers) do
  {:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
    IO.puts(body)
  {:ok, %HTTPoison.Response{status_code: status_code}} ->
    IO.puts("Received response with status code #{status_code}")
  {:error, %HTTPoison.Error{reason: reason}} ->
    IO.puts("HTTP request failed: #{reason}")
end

I’m able to make simple requests to other servers by altering the code slightly. I know the ip and authentication matches the working curl request, and go program.

Any advice is appreciated.

smathy · August 6, 2023, 7:55pm

No accept header?

al2o3cr · August 6, 2023, 8:08pm

That error comes from the Hackney machinery responsible for parsing the response; the fact that it’s going down the {:error, HTTPoison.Error} path means it’s failing to parse the opening line of the response - things like missing headers etc would generally result in an HTTPoison.Response with a status_code of 400.

The code in Hackney suggests some possiblities:

github.com

benoitc/hackney/blob/d6d075f9edcbb7e0b5c9418dba5dd34f2c95bc21/src/hackney_http.erl#L194-L227


      
          %% @doc parse status
          parse_response_line(#hparser{buffer=Buf}=St) ->
            case binary:split(Buf, <<"\r\n">>) of
              [Line, Rest] ->
                parse_response_version(Line, St#hparser{buffer=Rest});
              _ ->
                {error, bad_request}
            end.
          
          
          parse_response_version(<< "HTTP/", High, ".", Low, $\s, Rest/binary >>, St)
            when High >= $0, High =< $9, Low >= $0, Low =< $9 ->
            Version = { High -$0, Low - $0},
            parse_status(Rest, St, Version, <<>>);
          parse_response_version(_, _) ->
            {error, bad_request}.
          
          parse_status(<<>>, St, Version, Acc) ->
            parse_reason(<<>>, St, Version, Acc);
          parse_status(<< C, Rest/bits >>, St, Version, Acc) ->

This file has been truncated. show original

the server is sending too many blank lines before the HTTP/1.1 200 OK
the server is sending malformed line-endings which cause the binary:split on line 196 to not find two parts
the server is sending some other kind of mis-formatted value in the opening line (omitting the reason? Spelling HTTP wrong? Hard to tell)

The most reliable way to troubleshoot this is to capture the EXACT bytes the server is replying with, for instance with curl --trace:

Matts-MacBook-Pro-2:phoenix mattjones$ curl --trace - http://example.com
== Info: Rebuilt URL to: http://example.com/
== Info:   Trying 93.184.216.34...
== Info: TCP_NODELAY set
== Info: Connected to example.com (93.184.216.34) port 80 (#0)
=> Send header, 75 bytes (0x4b)
0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
0010: 48 6f 73 74 3a 20 65 78 61 6d 70 6c 65 2e 63 6f Host: example.co
0020: 6d 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 63 m..User-Agent: c
0030: 75 72 6c 2f 37 2e 35 34 2e 30 0d 0a 41 63 63 65 url/7.54.0..Acce
0040: 70 74 3a 20 2a 2f 2a 0d 0a 0d 0a                pt: */*....
<= Recv header, 17 bytes (0x11)
0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a                                              .
<= Recv header, 13 bytes (0xd)
0000: 41 67 65 3a 20 35 30 31 35 34 32 0d 0a          Age: 501542..
<= Recv header, 31 bytes (0x1f)
0000: 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c 3a 20 6d Cache-Control: m
0010: 61 78 2d 61 67 65 3d 36 30 34 38 30 30 0d 0a    ax-age=604800..
<= Recv header, 40 bytes (0x28)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te
0010: 78 74 2f 68 74 6d 6c 3b 20 63 68 61 72 73 65 74 xt/html; charset
0020: 3d 55 54 46 2d 38 0d 0a                         =UTF-8..
<= Recv header, 37 bytes (0x25)
0000: 44 61 74 65 3a 20 53 75 6e 2c 20 30 36 20 41 75 Date: Sun, 06 Au
0010: 67 20 32 30 32 33 20 32 30 3a 30 36 3a 35 35 20 g 2023 20:06:55 
0020: 47 4d 54 0d 0a                                  GMT..
<= Recv header, 26 bytes (0x1a)
...etcetcetc...

matthias_toepp · August 6, 2023, 8:51pm

Here is the output of curl —trace (showing only the part after the send header):

== Info: Mark bundle as not supporting multiuse
== Info: HTTP 1.0, assume close after body
<= Recv header, 16 bytes (0x10)
0000: 48 54 54 50 2f 31 2e 30 20 32 30 30 20 4f 4b 0a HTTP/1.0 200 OK.
<= Recv header, 24 bytes (0x18)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te
0010: 78 74 2f 68 74 6d 6c 0a                         xt/html.
<= Recv header, 1 bytes (0x1)
0000: 0a                                              .
<= Recv data, 502 bytes (0x1f6)
0000: 3c 68 74 6d 6c 3e 0a 3c 68 65 61 64 3e 0a 3c 6d <html>.<head>.<m
0010: 65 74 61 20 68 74 74 70 2d 65 71 75 69 76 3d 22 eta http-equiv="
0020: 63 6f 6e 74 65 6e 74 2d 6c 61 6e 67 75 61 67 65 content-language
0030: 22 20 63 6f 6e 74 65 6e 74 3d 22 65 6e 2d 75 73 " content="en-us
0040: 22 3e 0a 3c 6d 65 74 61 20 68 74 74 70 2d 65 71 ">.<meta http-eq
0050: 75 69 76 3d 22 63 6f 6e 74 65 6e 74 2d 74 79 70 uiv="content-typ
0060: 65 22 20 63 6f 6e 74 65 6e 74 3d 22 74 65 78 74 e" content="text
0070: 2f 68 74 6d 6c 3b 20 63 68 61 72 73 65 74 3d 77 /html; charset=w
0080: 69 6e 64 6f 77 73 2d 31 32 35 32 22 3e 0a 3c 6d indows-1252">.<m
0090: 65 74 61 20 68 74 74 70 2d 65 71 75 69 76 3d 22 eta http-equiv="
00a0: 70 72 61 67 6d 61 22 20 63 6f 6e 74 65 6e 74 3d pragma" content=
00b0: 22 6e 6f 2d 63 61 63 68 65 22 3e 0a 3c 74 69 74 "no-cache">.<tit
00c0: 6c 65 3e 72 61 77 20 64 61 74 61 3c 2f 74 69 74 le>raw data</tit
00d0: 6c 65 3e 0a 3c 2f 68 65 61 64 3e 0a 3c 62 6f 64 le>.</head>.<bod
00e0: 79 3e 0a 3c 70 3e 3c 66 6f 6e 74 20 66 61 63 65 y>.<p><font face
00f0: 3d 74 61 68 6f 6d 61 20 73 69 7a 65 3d 34 20 63 =tahoma size=4 c
0100: 6f 6c 6f 72 3d 23 46 46 30 30 30 30 3e 3c 62 3e olor=#FF0000><b>
0110: 72 61 77 20 64 61 74 61 3c 2f 62 3e 3c 2f 66 6f raw data</b></fo
0120: 6e 74 3e 3c 2f 70 3e 0a 3c 64 69 76 20 61 6c 69 nt></p>.<div ali
0130: 67 6e 3d 22 6c 65 66 74 22 3e 0a 3c 74 61 62 6c gn="left">.<tabl
0140: 65 20 62 6f 72 64 65 72 3d 22 30 22 20 63 65 6c e border="0" cel
0150: 6c 73 70 61 63 69 6e 67 3d 22 32 22 3e 0a 3c 74 lspacing="2">.<t
0160: 72 3e 0a 3c 74 64 3e 3c 66 6f 6e 74 20 66 61 63 r>.<td><font fac
0170: 65 3d 74 61 68 6f 6d 61 20 73 69 7a 65 3d 22 32 e=tahoma size="2
0180: 22 3e 3c 62 3e 4e 61 6d 65 26 6e 62 73 70 26 6e "><b>Name&nbsp&n
0190: 62 73 70 26 6e 62 73 70 3c 2f 62 3e 3c 2f 66 6f bsp&nbsp</b></fo
01a0: 6e 74 3e 3c 2f 74 64 3e 0a 3c 74 64 3e 3c 66 6f nt></td>.<td><fo
01b0: 6e 74 20 66 61 63 65 3d 74 61 68 6f 6d 61 20 73 nt face=tahoma s
01c0: 69 7a 65 3d 22 32 22 3e 3c 62 3e 56 61 6c 75 65 ize="2"><b>Value
01d0: 26 6e 62 73 70 26 6e 62 73 70 26 6e 62 73 70 3c &nbsp&nbsp&nbsp<
01e0: 2f 62 3e 3c 2f 66 6f 6e 74 3e 3c 2f 74 64 3e 0a /b></font></td>.
01f0: 3c 2f 74 72 3e 0a                               </tr>.
<html>
<head>
<meta http-equiv="content-language" content="en-us">
<meta http-equiv="content-type" content="text/html; charset=windows-1252">
<meta http-equiv="pragma" content="no-cache">
<title>raw data</title>
</head>
<body>
<p><font face=tahoma size=4 color=#FF0000><b>raw data</b></font></p>
<div align="left">
<table border="0" cellspacing="2">
<tr>
<td><font face=tahoma size="2"><b>Name&nbsp&nbsp&nbsp</b></font></td>
<td><font face=tahoma size="2"><b>Value&nbsp&nbsp&nbsp</b></font></td>
</tr>
<= Recv data, 367 bytes (0x16f)
0000: 3c 74 72 3e 0a 3c 74 64 3e 3c 66 6f 6e 74 20 66 <tr>.<td><font f
0010: 61 63 65 3d 74 61 68 6f 6d 61 20 73 69 7a 65 3d ace=tahoma size=
0020: 32 3e 6c 6f 77 5f 64 69 73 74 72 69 62 5f 70 72 2>low_distrib_pr
0030: 65 73 73 5f 61 6c 61 72 6d 26 6e 62 73 70 26 6e ess_alarm&nbsp&n
0040: 62 73 70 26 6e 62 73 70 3c 2f 66 6f 6e 74 3e 3c bsp&nbsp</font><
0050: 2f 74 64 3e 0a 3c 74 64 20 62 67 63 6f 6c 6f 72 /td>.<td bgcolor
0060: 3d 22 23 46 46 46 46 46 46 22 3e 3c 66 6f 6e 74 ="#FFFFFF"><font
0070: 20 66 61 63 65 3d 74 61 68 6f 6d 61 20 73 69 7a  face=tahoma siz
0080: 65 3d 32 20 63 6f 6c 6f 72 3d 23 30 30 30 30 30 e=2 color=#00000
0090: 30 3e 4f 66 66 26 6e 62 73 70 26 6e 62 73 70 26 0>Off&nbsp&nbsp&
00a0: 6e 62 73 70 3c 2f 66 6f 6e 74 3e 3c 2f 74 64 3e nbsp</font></td>
00b0: 0a 3c 2f 74 72 3e 0a 3c 74 72 3e 0a 3c 74 64 3e .</tr>.<tr>.<td>
00c0: 3c 66 6f 6e 74 20 66 61 63 65 3d 74 61 68 6f 6d <font face=tahom
00d0: 61 20 73 69 7a 65 3d 32 3e 66 6c 74 72 5f 68 69 a size=2>fltr_hi
00e0: 67 68 5f 65 66 66 5f 74 75 72 62 5f 61 6c 61 72 gh_eff_turb_alar
00f0: 6d 26 6e 62 73 70 26 6e 62 73 70 26 6e 62 73 70 m&nbsp&nbsp&nbsp
0100: 3c 2f 66 6f 6e 74 3e 3c 2f 74 64 3e 0a 3c 74 64 </font></td>.<td
0110: 20 62 67 63 6f 6c 6f 72 3d 22 23 46 46 46 46 46  bgcolor="#FFFFF
0120: 46 22 3e 3c 66 6f 6e 74 20 66 61 63 65 3d 74 61 F"><font face=ta
0130: 68 6f 6d 61 20 73 69 7a 65 3d 32 20 63 6f 6c 6f homa size=2 colo
0140: 72 3d 23 30 30 30 30 30 30 3e 4f 66 66 26 6e 62 r=#000000>Off&nb
0150: 73 70 26 6e 62 73 70 26 6e 62 73 70 3c 2f 66 6f sp&nbsp&nbsp</fo
0160: 6e 74 3e 3c 2f 74 64 3e 0a 3c 2f 74 72 3e 0a    nt></td>.</tr>.
<tr>
<td><font face=tahoma size=2>low_distrib_press_alarm&nbsp&nbsp&nbsp</font></td>
<td bgcolor="#FFFFFF"><font face=tahoma size=2 color=#000000>Off&nbsp&nbsp&nbsp</font></td>
</tr>
<tr>
<td><font face=tahoma size=2>fltr_high_eff_turb_alarm&nbsp&nbsp&nbsp</font></td>
<td bgcolor="#FFFFFF"><font face=tahoma size=2 color=#000000>Off&nbsp&nbsp&nbsp</font></td>
</tr>
<= Recv data, 182 bytes (0xb6)
0000: 3c 74 72 3e 0a 3c 74 64 3e 3c 66 6f 6e 74 20 66 <tr>.<td><font f
0010: 61 63 65 3d 74 61 68 6f 6d 61 20 73 69 7a 65 3d ace=tahoma size=
0020: 32 3e 66 6c 74 5f 68 69 67 68 5f 63 6c 65 61 72 2>flt_high_clear
0030: 77 65 6c 6c 5f 6c 76 6c 26 6e 62 73 70 26 6e 62 well_lvl&nbsp&nb
0040: 73 70 26 6e 62 73 70 3c 2f 66 6f 6e 74 3e 3c 2f sp&nbsp</font></
0050: 74 64 3e 0a 3c 74 64 20 62 67 63 6f 6c 6f 72 3d td>.<td bgcolor=
0060: 22 23 46 46 46 46 46 46 22 3e 3c 66 6f 6e 74 20 "#FFFFFF"><font
0070: 66 61 63 65 3d 74 61 68 6f 6d 61 20 73 69 7a 65 face=tahoma size
0080: 3d 32 20 63 6f 6c 6f 72 3d 23 30 30 30 30 30 30 =2 color=#000000
0090: 3e 4f 66 66 26 6e 62 73 70 26 6e 62 73 70 26 6e >Off&nbsp&nbsp&n
00a0: 62 73 70 3c 2f 66 6f 6e 74 3e 3c 2f 74 64 3e 0a bsp</font></td>.
00b0: 3c 2f 74 72 3e 0a                               </tr>.
<tr>
<td><font face=tahoma size=2>flt_high_clearwell_lvl&nbsp&nbsp&nbsp</font></td>
<td bgcolor="#FFFFFF"><font face=tahoma size=2 color=#000000>Off&nbsp&nbsp&nbsp</font></td>
</tr>
<= Recv data, 16 bytes (0x10)
0000: 3c 2f 74 61 62 6c 65 3e 0a 3c 2f 64 69 76 3e 0a </table>.</div>.
</table>
</div>
<= Recv data, 77 bytes (0x4d)
0000: 3c 70 3e 3c 66 6f 6e 74 20 66 61 63 65 3d 74 61 <p><font face=ta
0010: 68 6f 6d 61 20 73 69 7a 65 3d 32 3e 3c 62 3e 3c homa size=2><b><
0020: 61 20 68 72 65 66 3d 2e 2f 64 65 66 61 75 6c 74 a href=./default
0030: 2e 68 74 6d 3e 42 41 43 4b 3c 2f 61 3e 3c 2f 62 .htm>BACK</a></b
0040: 3e 3c 2f 66 6f 6e 74 3e 3c 2f 70 3e 0a          ></font></p>.
<p><font face=tahoma size=2><b><a href=./default.htm>BACK</a></b></font></p>
<= Recv data, 104 bytes (0x68)
0000: 3c 70 3e 3c 66 6f 6e 74 20 66 61 63 65 3d 74 61 <p><font face=ta
0010: 68 6f 6d 61 20 73 69 7a 65 3d 31 3e a9 20 31 39 homa size=1>. 19
0020: 39 33 2d 32 30 30 38 20 3c 61 20 68 72 65 66 3d 93-2008 <a href=
0030: 68 74 74 70 3a 2f 2f 77 77 77 2e 72 65 64 6c 69 http://www.redli
0040: 6f 6e 2e 6e 65 74 3e 52 65 64 20 4c 69 6f 6e 20 on.net>Red Lion
0050: 43 6f 6e 74 72 6f 6c 73 3c 2f 61 3e 3c 2f 66 6f Controls</a></fo
0060: 6e 74 3e 3c 2f 70 3e 0a                         nt></p>.
<p><font face=tahoma size=1>� 1993-2008 <a href=http://www.redlion.net>Red Lion Controls</a></font></p>
<= Recv data, 16 bytes (0x10)
0000: 3c 2f 62 6f 64 79 3e 0a 3c 2f 68 74 6d 6c 3e 0a </body>.</html>.
</body>
</html>
== Info: Closing connection 0

al2o3cr · August 6, 2023, 10:21pm

curl appears to tolerate just sending \n, but Hackney will not. The spec requires \r\n.

matthias_toepp · August 7, 2023, 9:16pm

(deleted)

matthias_toepp · August 7, 2023, 9:14pm

It’s kind of marvelous to me that you were able to figure that out. Thanks for the insight. I guess I’ll try a different http library then.

Maybe I’ll post this as an issue for Hackney. It kind of makes sense as apparently it’s following the spec which you linked to, but on the other hand, from my perspective, curl and go ‘worked’, in terms of getting the job done and httpoison didn’t.

matthias_toepp · August 8, 2023, 8:29pm

I opened an issue relating to this on hackney’s github page:

github.com/benoitc/hackney

GET request with basic authorization -- "HTTP request failed: bad_request"

opened 08:24PM - 08 Aug 23 UTC

human154

I’m trying to get html from an old machine. I’m able to successfully get a re…sponse on the command line with the following curl command (and I have a go program working as well): ```bash curl 200.200.200.200/aaa/001 --user "user:password"` ``` Using Httpoison I get the error: “HTTP request failed: bad_request”: ```elixir HTTPoison.start() username = "user" password = "password" credentials_encoded = Base.encode64("#{username}:#{password}") headers = [{"Authorization", "Basic #{credentials_encoded}"}] case HTTPoison.get("200.200.200.200/aaa/001", headers) do {:ok, %HTTPoison.Response{status_code: 200, body: body}} -> IO.puts(body) {:ok, %HTTPoison.Response{status_code: status_code}} -> IO.puts("Received response with status code #{status_code}") {:error, %HTTPoison.Error{reason: reason}} -> IO.puts("HTTP request failed: #{reason}") end ``` I asked about this on the elixir forum (https://elixirforum.com/t/httpoison-get-request-with-basic-authorization-http-request-failed-bad-request/57470/3) and was told that it is likely because the response contains '\n' instead of '\r\n': ``` 0000: 48 54 54 50 2f 31 2e 30 20 32 30 30 20 4f 4b 0a HTTP/1.0 200 OK. curl appears to tolerate just sending \n, but Hackney will not. The [spec 2](https://www.w3.org/Protocols/HTTP/1.0/spec#Response) requires \r\n. ``` My understanding being that '0a', the last of the hex values indicates a LF (line feed) value and the spec calls for CR (carriage returen hex '0d') and LF. If this is the issue which causes the error message.. even if hackney is "following the spec", from my perspective as a user, curl and go just ‘worked’ with no issue, in terms of getting the job done for me, while I seem to have no way to move forward in elixir with httpoison. Any feedback is appreciated. Thanks!

matthias_toepp · October 19, 2023, 1:21pm

Just for the record…I’m told by the maintainers that this issue is now resolved!