matthias_toepp

matthias_toepp

I’m trying to get html from an old machine.

I’m able to successfully get a response on the command line with this (and I have a go program working as well):

curl 200.200.200.200/aaa/001 --user "user:password"

I get the error: “HTTP request failed: bad_request” when trying to do that with an Elixir script:

HTTPoison.start()

username = "user"
password = "password"

credentials_encoded = Base.encode64("#{username}:#{password}")

headers = [{"Authorization", "Basic #{credentials_encoded}"}]

case HTTPoison.get("200.200.200.200/aaa/001", headers) do
  {:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
    IO.puts(body)
  {:ok, %HTTPoison.Response{status_code: status_code}} ->
    IO.puts("Received response with status code #{status_code}")
  {:error, %HTTPoison.Error{reason: reason}} ->
    IO.puts("HTTP request failed: #{reason}")
end

I’m able to make simple requests to other servers by altering the code slightly. I know the ip and authentication matches the working curl request, and go program.

Any advice is appreciated.

Showing Posts 1 to 8

smathy

smathy

No accept header?

al2o3cr

al2o3cr

That error comes from the Hackney machinery responsible for parsing the response; the fact that it’s going down the {:error, HTTPoison.Error} path means it’s failing to parse the opening line of the response - things like missing headers etc would generally result in an HTTPoison.Response with a status_code of 400.

The code in Hackney suggests some possiblities:

https://github.com/benoitc/hackney/blob/d6d075f9edcbb7e0b5c9418dba5dd34f2c95bc21/src/hackney_http.erl#L194-L227

  • the server is sending too many blank lines before the HTTP/1.1 200 OK
  • the server is sending malformed line-endings which cause the binary:split on line 196 to not find two parts
  • the server is sending some other kind of mis-formatted value in the opening line (omitting the reason? Spelling HTTP wrong? Hard to tell)

The most reliable way to troubleshoot this is to capture the EXACT bytes the server is replying with, for instance with curl --trace:

Matts-MacBook-Pro-2:phoenix mattjones$ curl --trace - http://example.com
== Info: Rebuilt URL to: http://example.com/
== Info:   Trying 93.184.216.34...
== Info: TCP_NODELAY set
== Info: Connected to example.com (93.184.216.34) port 80 (#0)
=> Send header, 75 bytes (0x4b)
0000: 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a GET / HTTP/1.1..
0010: 48 6f 73 74 3a 20 65 78 61 6d 70 6c 65 2e 63 6f Host: example.co
0020: 6d 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20 63 m..User-Agent: c
0030: 75 72 6c 2f 37 2e 35 34 2e 30 0d 0a 41 63 63 65 url/7.54.0..Acce
0040: 70 74 3a 20 2a 2f 2a 0d 0a 0d 0a                pt: */*....
<= Recv header, 17 bytes (0x11)
0000: 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010: 0a                                              .
<= Recv header, 13 bytes (0xd)
0000: 41 67 65 3a 20 35 30 31 35 34 32 0d 0a          Age: 501542..
<= Recv header, 31 bytes (0x1f)
0000: 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c 3a 20 6d Cache-Control: m
0010: 61 78 2d 61 67 65 3d 36 30 34 38 30 30 0d 0a    ax-age=604800..
<= Recv header, 40 bytes (0x28)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te
0010: 78 74 2f 68 74 6d 6c 3b 20 63 68 61 72 73 65 74 xt/html; charset
0020: 3d 55 54 46 2d 38 0d 0a                         =UTF-8..
<= Recv header, 37 bytes (0x25)
0000: 44 61 74 65 3a 20 53 75 6e 2c 20 30 36 20 41 75 Date: Sun, 06 Au
0010: 67 20 32 30 32 33 20 32 30 3a 30 36 3a 35 35 20 g 2023 20:06:55 
0020: 47 4d 54 0d 0a                                  GMT..
<= Recv header, 26 bytes (0x1a)
...etcetcetc...
matthias_toepp

matthias_toepp OP

Here is the output of curl —trace (showing only the part after the send header):

== Info: Mark bundle as not supporting multiuse
== Info: HTTP 1.0, assume close after body
<= Recv header, 16 bytes (0x10)
0000: 48 54 54 50 2f 31 2e 30 20 32 30 30 20 4f 4b 0a HTTP/1.0 200 OK.
<= Recv header, 24 bytes (0x18)
0000: 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te
0010: 78 74 2f 68 74 6d 6c 0a                         xt/html.
<= Recv header, 1 bytes (0x1)
0000: 0a                                              .
<= Recv data, 502 bytes (0x1f6)
0000: 3c 68 74 6d 6c 3e 0a 3c 68 65 61 64 3e 0a 3c 6d <html>.<head>.<m
0010: 65 74 61 20 68 74 74 70 2d 65 71 75 69 76 3d 22 eta http-equiv="
0020: 63 6f 6e 74 65 6e 74 2d 6c 61 6e 67 75 61 67 65 content-language
0030: 22 20 63 6f 6e 74 65 6e 74 3d 22 65 6e 2d 75 73 " content="en-us
0040: 22 3e 0a 3c 6d 65 74 61 20 68 74 74 70 2d 65 71 ">.<meta http-eq
0050: 75 69 76 3d 22 63 6f 6e 74 65 6e 74 2d 74 79 70 uiv="content-typ
0060: 65 22 20 63 6f 6e 74 65 6e 74 3d 22 74 65 78 74 e" content="text
0070: 2f 68 74 6d 6c 3b 20 63 68 61 72 73 65 74 3d 77 /html; charset=w
0080: 69 6e 64 6f 77 73 2d 31 32 35 32 22 3e 0a 3c 6d indows-1252">.<m
0090: 65 74 61 20 68 74 74 70 2d 65 71 75 69 76 3d 22 eta http-equiv="
00a0: 70 72 61 67 6d 61 22 20 63 6f 6e 74 65 6e 74 3d pragma" content=
00b0: 22 6e 6f 2d 63 61 63 68 65 22 3e 0a 3c 74 69 74 "no-cache">.<tit
00c0: 6c 65 3e 72 61 77 20 64 61 74 61 3c 2f 74 69 74 le>raw data</tit
00d0: 6c 65 3e 0a 3c 2f 68 65 61 64 3e 0a 3c 62 6f 64 le>.</head>.<bod
00e0: 79 3e 0a 3c 70 3e 3c 66 6f 6e 74 20 66 61 63 65 y>.<p><font face
00f0: 3d 74 61 68 6f 6d 61 20 73 69 7a 65 3d 34 20 63 =tahoma size=4 c
0100: 6f 6c 6f 72 3d 23 46 46 30 30 30 30 3e 3c 62 3e olor=#FF0000><b>
0110: 72 61 77 20 64 61 74 61 3c 2f 62 3e 3c 2f 66 6f raw data</b></fo
0120: 6e 74 3e 3c 2f 70 3e 0a 3c 64 69 76 20 61 6c 69 nt></p>.<div ali
0130: 67 6e 3d 22 6c 65 66 74 22 3e 0a 3c 74 61 62 6c gn="left">.<tabl
0140: 65 20 62 6f 72 64 65 72 3d 22 30 22 20 63 65 6c e border="0" cel
0150: 6c 73 70 61 63 69 6e 67 3d 22 32 22 3e 0a 3c 74 lspacing="2">.<t
0160: 72 3e 0a 3c 74 64 3e 3c 66 6f 6e 74 20 66 61 63 r>.<td><font fac
0170: 65 3d 74 61 68 6f 6d 61 20 73 69 7a 65 3d 22 32 e=tahoma size="2
0180: 22 3e 3c 62 3e 4e 61 6d 65 26 6e 62 73 70 26 6e "><b>Name&nbsp&n
0190: 62 73 70 26 6e 62 73 70 3c 2f 62 3e 3c 2f 66 6f bsp&nbsp</b></fo
01a0: 6e 74 3e 3c 2f 74 64 3e 0a 3c 74 64 3e 3c 66 6f nt></td>.<td><fo
01b0: 6e 74 20 66 61 63 65 3d 74 61 68 6f 6d 61 20 73 nt face=tahoma s
01c0: 69 7a 65 3d 22 32 22 3e 3c 62 3e 56 61 6c 75 65 ize="2"><b>Value
01d0: 26 6e 62 73 70 26 6e 62 73 70 26 6e 62 73 70 3c &nbsp&nbsp&nbsp<
01e0: 2f 62 3e 3c 2f 66 6f 6e 74 3e 3c 2f 74 64 3e 0a /b></font></td>.
01f0: 3c 2f 74 72 3e 0a                               </tr>.
<html>
<head>
<meta http-equiv="content-language" content="en-us">
<meta http-equiv="content-type" content="text/html; charset=windows-1252">
<meta http-equiv="pragma" content="no-cache">
<title>raw data</title>
</head>
<body>
<p><font face=tahoma size=4 color=#FF0000><b>raw data</b></font></p>
<div align="left">
<table border="0" cellspacing="2">
<tr>
<td><font face=tahoma size="2"><b>Name&nbsp&nbsp&nbsp</b></font></td>
<td><font face=tahoma size="2"><b>Value&nbsp&nbsp&nbsp</b></font></td>
</tr>
<= Recv data, 367 bytes (0x16f)
0000: 3c 74 72 3e 0a 3c 74 64 3e 3c 66 6f 6e 74 20 66 <tr>.<td><font f
0010: 61 63 65 3d 74 61 68 6f 6d 61 20 73 69 7a 65 3d ace=tahoma size=
0020: 32 3e 6c 6f 77 5f 64 69 73 74 72 69 62 5f 70 72 2>low_distrib_pr
0030: 65 73 73 5f 61 6c 61 72 6d 26 6e 62 73 70 26 6e ess_alarm&nbsp&n
0040: 62 73 70 26 6e 62 73 70 3c 2f 66 6f 6e 74 3e 3c bsp&nbsp</font><
0050: 2f 74 64 3e 0a 3c 74 64 20 62 67 63 6f 6c 6f 72 /td>.<td bgcolor
0060: 3d 22 23 46 46 46 46 46 46 22 3e 3c 66 6f 6e 74 ="#FFFFFF"><font
0070: 20 66 61 63 65 3d 74 61 68 6f 6d 61 20 73 69 7a  face=tahoma siz
0080: 65 3d 32 20 63 6f 6c 6f 72 3d 23 30 30 30 30 30 e=2 color=#00000
0090: 30 3e 4f 66 66 26 6e 62 73 70 26 6e 62 73 70 26 0>Off&nbsp&nbsp&
00a0: 6e 62 73 70 3c 2f 66 6f 6e 74 3e 3c 2f 74 64 3e nbsp</font></td>
00b0: 0a 3c 2f 74 72 3e 0a 3c 74 72 3e 0a 3c 74 64 3e .</tr>.<tr>.<td>
00c0: 3c 66 6f 6e 74 20 66 61 63 65 3d 74 61 68 6f 6d <font face=tahom
00d0: 61 20 73 69 7a 65 3d 32 3e 66 6c 74 72 5f 68 69 a size=2>fltr_hi
00e0: 67 68 5f 65 66 66 5f 74 75 72 62 5f 61 6c 61 72 gh_eff_turb_alar
00f0: 6d 26 6e 62 73 70 26 6e 62 73 70 26 6e 62 73 70 m&nbsp&nbsp&nbsp
0100: 3c 2f 66 6f 6e 74 3e 3c 2f 74 64 3e 0a 3c 74 64 </font></td>.<td
0110: 20 62 67 63 6f 6c 6f 72 3d 22 23 46 46 46 46 46  bgcolor="#FFFFF
0120: 46 22 3e 3c 66 6f 6e 74 20 66 61 63 65 3d 74 61 F"><font face=ta
0130: 68 6f 6d 61 20 73 69 7a 65 3d 32 20 63 6f 6c 6f homa size=2 colo
0140: 72 3d 23 30 30 30 30 30 30 3e 4f 66 66 26 6e 62 r=#000000>Off&nb
0150: 73 70 26 6e 62 73 70 26 6e 62 73 70 3c 2f 66 6f sp&nbsp&nbsp</fo
0160: 6e 74 3e 3c 2f 74 64 3e 0a 3c 2f 74 72 3e 0a    nt></td>.</tr>.
<tr>
<td><font face=tahoma size=2>low_distrib_press_alarm&nbsp&nbsp&nbsp</font></td>
<td bgcolor="#FFFFFF"><font face=tahoma size=2 color=#000000>Off&nbsp&nbsp&nbsp</font></td>
</tr>
<tr>
<td><font face=tahoma size=2>fltr_high_eff_turb_alarm&nbsp&nbsp&nbsp</font></td>
<td bgcolor="#FFFFFF"><font face=tahoma size=2 color=#000000>Off&nbsp&nbsp&nbsp</font></td>
</tr>
<= Recv data, 182 bytes (0xb6)
0000: 3c 74 72 3e 0a 3c 74 64 3e 3c 66 6f 6e 74 20 66 <tr>.<td><font f
0010: 61 63 65 3d 74 61 68 6f 6d 61 20 73 69 7a 65 3d ace=tahoma size=
0020: 32 3e 66 6c 74 5f 68 69 67 68 5f 63 6c 65 61 72 2>flt_high_clear
0030: 77 65 6c 6c 5f 6c 76 6c 26 6e 62 73 70 26 6e 62 well_lvl&nbsp&nb
0040: 73 70 26 6e 62 73 70 3c 2f 66 6f 6e 74 3e 3c 2f sp&nbsp</font></
0050: 74 64 3e 0a 3c 74 64 20 62 67 63 6f 6c 6f 72 3d td>.<td bgcolor=
0060: 22 23 46 46 46 46 46 46 22 3e 3c 66 6f 6e 74 20 "#FFFFFF"><font
0070: 66 61 63 65 3d 74 61 68 6f 6d 61 20 73 69 7a 65 face=tahoma size
0080: 3d 32 20 63 6f 6c 6f 72 3d 23 30 30 30 30 30 30 =2 color=#000000
0090: 3e 4f 66 66 26 6e 62 73 70 26 6e 62 73 70 26 6e >Off&nbsp&nbsp&n
00a0: 62 73 70 3c 2f 66 6f 6e 74 3e 3c 2f 74 64 3e 0a bsp</font></td>.
00b0: 3c 2f 74 72 3e 0a                               </tr>.
<tr>
<td><font face=tahoma size=2>flt_high_clearwell_lvl&nbsp&nbsp&nbsp</font></td>
<td bgcolor="#FFFFFF"><font face=tahoma size=2 color=#000000>Off&nbsp&nbsp&nbsp</font></td>
</tr>
<= Recv data, 16 bytes (0x10)
0000: 3c 2f 74 61 62 6c 65 3e 0a 3c 2f 64 69 76 3e 0a </table>.</div>.
</table>
</div>
<= Recv data, 77 bytes (0x4d)
0000: 3c 70 3e 3c 66 6f 6e 74 20 66 61 63 65 3d 74 61 <p><font face=ta
0010: 68 6f 6d 61 20 73 69 7a 65 3d 32 3e 3c 62 3e 3c homa size=2><b><
0020: 61 20 68 72 65 66 3d 2e 2f 64 65 66 61 75 6c 74 a href=./default
0030: 2e 68 74 6d 3e 42 41 43 4b 3c 2f 61 3e 3c 2f 62 .htm>BACK</a></b
0040: 3e 3c 2f 66 6f 6e 74 3e 3c 2f 70 3e 0a          ></font></p>.
<p><font face=tahoma size=2><b><a href=./default.htm>BACK</a></b></font></p>
<= Recv data, 104 bytes (0x68)
0000: 3c 70 3e 3c 66 6f 6e 74 20 66 61 63 65 3d 74 61 <p><font face=ta
0010: 68 6f 6d 61 20 73 69 7a 65 3d 31 3e a9 20 31 39 homa size=1>. 19
0020: 39 33 2d 32 30 30 38 20 3c 61 20 68 72 65 66 3d 93-2008 <a href=
0030: 68 74 74 70 3a 2f 2f 77 77 77 2e 72 65 64 6c 69 http://www.redli
0040: 6f 6e 2e 6e 65 74 3e 52 65 64 20 4c 69 6f 6e 20 on.net>Red Lion
0050: 43 6f 6e 74 72 6f 6c 73 3c 2f 61 3e 3c 2f 66 6f Controls</a></fo
0060: 6e 74 3e 3c 2f 70 3e 0a                         nt></p>.
<p><font face=tahoma size=1>� 1993-2008 <a href=http://www.redlion.net>Red Lion Controls</a></font></p>
<= Recv data, 16 bytes (0x10)
0000: 3c 2f 62 6f 64 79 3e 0a 3c 2f 68 74 6d 6c 3e 0a </body>.</html>.
</body>
</html>
== Info: Closing connection 0
al2o3cr

al2o3cr

curl appears to tolerate just sending \n, but Hackney will not. The spec requires \r\n.

matthias_toepp

matthias_toepp OP

(deleted)

matthias_toepp

matthias_toepp OP

It’s kind of marvelous to me that you were able to figure that out. Thanks for the insight. I guess I’ll try a different http library then.

Maybe I’ll post this as an issue for Hackney. It kind of makes sense as apparently it’s following the spec which you linked to, but on the other hand, from my perspective, curl and go ‘worked’, in terms of getting the job done and httpoison didn’t.

matthias_toepp

matthias_toepp OP

I opened an issue relating to this on hackney’s github page:

https://github.com/benoitc/hackney/issues/723

matthias_toepp

matthias_toepp OP

Just for the record…I’m told by the maintainers that this issue is now resolved!

— All posts loaded —

Where Next? Top

Trending in Questions Top

water
I’m aware of 2 such projects, are there others? git: GitHub - team-alembic/realworld: A fullstack Phoenix LiveView application with back...
New
katta
I having some trouble figuring out if I have set myself too strict of standards for my production server. Currently I can handle 75% of r...
New
peck
Hi everyone. My team and I have been working on a fairly modest app based around video streaming and chat, but we’ve landed a customer t...
New
bradley
I really like the adapter patterns that ecto, nebulex, waffle, etc. use and would love find something similar for a key management servic...
New
unaware8150
Hello folks! So at work, we are seeing some situations where we have to define some “fixed” strings that are used across the codebase in...
New
Alvinkariuki
How Can I Optimise Compile Time Dependencies I have been building an elixir application for about 2 years now. Many modules and files ha...
New

Other Trending Topics Top

budgie
A little off-topic, but I feel like people here have a good head on their shoulders. I used to be quite good at making software. Was luc...
New
KristerV
Hey. Is there anyone here who creates agents in their apps? Not talking about using agents, but creating them. I’m finding it pretty diff...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
juhalehtonen
There has been a thread to discuss the Stack Overflow Developer Survey on this forum every year since 2018, so here’s yet another one for...
New
budgie
I love Elixir. It’s one of 2 programming languages I’ve ever fallen in love with. But I don’t use it anymore. Serverless was the promis...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews