HTTPoison multipart post

what’s the error you are getting @pramsky ?

The vendor endpoint returns an error 400.

To debug, I did the 2 posts to pipedream and see there is a difference in the content type used for the form fields.

The curl request

01 - curl request294×152 9.32 KB

Httpoison request

02 - elixir httpoison request816×198 18.3 KB

Can you extract the actual contents of the two requests?

Here are the responses when sent to httpbin.org/anything. Converted to elixir map.

This is the json response from the curl request

%{
  "args" => %{},
  "data" => "",
  "files" => %{"file" => "<<BASEENCODED IMAGE FILE>>"},
  "form" => %{
    "from" => "5555555555",
    "orgid" => "1234567890987654321",
    "to" => "1234"
  },
  "headers" => %{
    "Accept" => "*/*",
    "Authorization" => "Bearer asupersecretkey",
    "Content-Length" => "258300",
    "Content-Type" => "multipart/form-data; boundary=------------------------ef42181273f52b98",
    "Host" => "httpbin.org",
    "User-Agent" => "curl/7.79.1",
    "X-Amzn-Trace-Id" => "Root=1-62e052b9-5df66f172fe5904b220380f3"
  },
  "json" => nil,
  "method" => "POST",
  "origin" => "my.ip.ad.dr",
  "url" => "https://httpbin.org/anything"
}

This is the response from HTTPoison

%{
  "args" => %{},
  "data" => "",
  "files" => %{"file" => "<<BASEENCODED IMAGE FILE>>"},
  "form" => %{
    "from" => "5555555555",
    "orgid" => "1234567890987654321",
    "to" => "1234"
  },
  "headers" => %{
    "Authorization" => "Bearer asupersecretkey",
    "Content-Length" => "258518",
    "Content-Type" => "multipart/form-data; boundary=---------------------------jtlqvenjbyxkxrrs",
    "Host" => "httpbin.org",
    "User-Agent" => "hackney/1.18.1",
    "X-Amzn-Trace-Id" => "Root=1-62e051d5-6dae5f9e5b2a020d6ad368dd"
  },
  "json" => nil,
  "method" => "POST",
  "origin" => "my.ip.ad.dr",
  "url" => "https://httpbin.org/anything"
}

I don’t think comparing what comes out of HTTPBin’s parser is sufficient to see the issue - I only see two differences:

• the boundary string in the Content-Type; that’s intentionally random and so not relevant
• the Accept header from Curl is not present in the other

It’s worth a try to add that header; some HTTP implementations are more selective about the headers they expect.

Otherwise you’ll need the actual bytes being sent in the two requests - there’s evidently some difference, since Pipedream shows a slightly different shape, despite HTTPBin seeing the same values.

I was able to get the post text fields to show up in pipe dream as text by adding [“Content-Type”: “text/plain”] to the payload tuple.

It still does not work on the vendor site. I reached out to the vendor and he suggested it may be the boundary that is causing the issue. I have the logs of the POST request from the vendor.

The working request via Insomnia REST Client

28/07 16.57.55.545       (2083 ms) 23.252.62.194        POST /api HTTP/1.1
content-type: [multipart/form-data; boundary=X-INSOMNIA-BOUNDARY]
accept: [*/*]
cookie: [sessionId=api_1.2.3.194_1659022119229_-1260159572]
authorization: [Bearer supersecretauthtoken]
host: [example.net]
content-length: [258197]
user-agent: [insomnia/2022.4.2]

file=/opt/stserver/tmp/foo.png&to=1234&from=15555555555&orgid=1234567890987564321

HTTP/1.1 200 OK
Date: [Thu, 28 Jul 2022 16:57:55]
Access-Control-Allow-Origin: [*]
Access-Control-Allow-Credentials: [true]
Server: [Smile CTI Server]
Set-Cookie: [sessionId=api_1.2.3.194_1659027473462_-1149585348]
Connection: [close]
Content-Type: [application/json; charset=UTF-8]
Content-Length: [59]

The non working HTTPoison request

28/07 17.03.09.660       (578 ms)1.2.3.194 POST /api HTTP/1.1
content-type: [multipart/form-data; boundary=---------------------------nhzzivvxhvzlhrmf]
accept: [*/*]
authorization: [Bearer supersecretauthtoken]
host: [example.net]
content-length: [258478]
user-agent: [hackney/1.18.1]


HTTP/1.1 400 Bad Request
Date: [Thu, 28 Jul 2022 17:03:09]
Access-Control-Allow-Origin: [*]
Access-Control-Allow-Credentials: [true]
Server: [Smile CTI Server]
Connection: [close]
Content-Length: [0]

The boundary from HTTPoison meets the requirements of RFC2046 - it’s more than 1 and less than 70 characters from the specified character set. If the vendor’s parser is breaking on it, the vendor’s parser is broken.

These logs show that the vendor’s parser might be broken (there’s no data printed from the second one) but they offer no suggestion of what the problem is. The only thing that can tell us what’s going on is the raw bytes actually sent to the server.

One way to capture these is with netcat (frequently shortened to nc) - a handy utility program that will listen on a specified port and then print exactly what comes in when invoked with -l

For curl:

Matts-MacBook-Pro-2:~ mattjones$ nc -l localhost 8888
POST / HTTP/1.1
Host: localhost:8888
User-Agent: curl/7.54.0
Accept: */*
Authorization: Bearer asupersecrettokentoauthenticatewith
Content-Length: 619
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------58692fd24caf27b7

--------------------------58692fd24caf27b7
Content-Disposition: form-data; name="orgid"

1234567890987654321
--------------------------58692fd24caf27b7
Content-Disposition: form-data; name="from"

5555555555
--------------------------58692fd24caf27b7
Content-Disposition: form-data; name="to"

1234
--------------------------58692fd24caf27b7
Content-Disposition: form-data; name="file"; filename="helloworld.erl"
Content-Type: application/octet-stream

% hello world program
-module(helloworld).
-export([start/0]).

start() ->
    spawn(fun() -> ok end).
--------------------------58692fd24caf27b7--

Versus the result from HTTPoison:

Matts-MacBook-Pro-2:~ mattjones$ nc -l 127.0.0.1 8888
POST / HTTP/1.1
Authorization: Bearer asupersecrettokentoauthenticatewith
Host: 127.0.0.1:8888
User-Agent: hackney/1.18.1
Content-Type: multipart/form-data; boundary=---------------------------bydneyeiuypqrgyb
Content-Length: 834

-----------------------------bydneyeiuypqrgyb
content-length: 102
content-type: application/octet-stream
content-disposition: form-data; name="file"; filename="helloworld.erl"

% hello world program
-module(helloworld).
-export([start/0]).

start() ->
    spawn(fun() -> ok end).
-----------------------------bydneyeiuypqrgyb
content-length: 10
content-type: application/octet-stream
content-disposition: form-data; name="from"

5555555555
-----------------------------bydneyeiuypqrgyb
content-length: 4
content-type: application/octet-stream
content-disposition: form-data; name="to"

1234
-----------------------------bydneyeiuypqrgyb
content-length: 19
content-type: application/octet-stream
content-disposition: form-data; name="orgid"

1234567890987654321
-----------------------------bydneyeiuypqrgyb--

Notable differences:

• HTTPoison always sends Content-Length headers
• HTTPoison always sends Content-Type headers

You could try passing an explicit Content-Type to the form fields; there are bug reports that suggest that some servers don’t handle application/octet-stream fields correctly. For instance, pass {"from", from, [{"content-type", "text/plain"}]} instead of {"from", from}. Doing this locally has the desired effect (the content-type values change for the form fields).

On the other hand, apparently some servers don’t like Content-Length:

https://github.com/edgurgel/httpoison/issues/286

Thanks for the response @al2o3cr

I believe you may be right about the content-length. I setup a local server to capture raw post dumps and could not really see why the httpoison request would fail while the insomnia request works fine as both requests looked correct. Testing it with pipedream showed no errors and I was able to download the attached file.

If some servers don’t like content length on the fields, that would explain why I get a request 400 from this vendor. I reached out to them asking about it.

@al2o3cr

The vendor made a fix on their end and the posts are working without issues. Thanks for your help!