Trouble submitting data/file to Kraken.io API using HTTPoison

Questions / Help
1

Afternoon everyone!

I’ve been scratching my head over this one all morning. As always, a simple task has led me down the rabbit hole! :joy:

I’m trying to submit an image to the Kraken API for image processing (although if anyone has an alternative to image optimisation — I’m all ears!)

The following curl request does exactly what I need it to (I’ve omitted my API keys):

curl https://api.kraken.io/v1/upload -X POST --form data='{"auth": {"api_key": "API_KEY", "api_secret": "API_SECRET"}, "wait":true, "dev":true}' --form upload=@example.jpg

I’ve been trying to accomplish that from Elixir, using HTTPoison. Following all the troubleshooting issues/thread scattered around online, I think the closest I’ve got is this:

HTTPoison.post!("https://api.kraken.io/v1/upload", {:multipart, [{:upload, "~/example.jpg", { ["data"], [auth: [api_key: "\"API_KEY\"", api_secret: "\"API_SECRET\""], wait: "true", dev: "true"]}}]})

Perhaps this is just a syntax issue, as hackney puts out a variety of errors depending on how I tweak things, but the general gist is that it either can’t map over a :list or that there’s not a function clause that matches. Is there anything blindingly obviously wrong with what I’ve written. I’m still quite “green” with Elixir, and I’ve ended up elbow deep in Erlang syntax, so could be missing something simple.

Alternatively, does anyone have a good solution to optimising images (and creating various sizes) before placing them all on S3. I’ve previously put a master image in storage and processed alterations on the fly with services like Imgix, but want to move away from that because of the expense mainly.

Any help would be greatly appreciated!

Cheers, Jamie.

2

I think your HTTPoison options are incorrect. Looking at the hackney options, it looks like you need a {:file, "~/example.jpg"} rather than {:upload, ...}. You can also see it referenced in the HTTPoison docblock for request().

3

There is this Elixir wrapper for ImageMagick https://github.com/route/mogrify for image processing.

4

Right, I see. In my head I was mapping the “upload” param from the curl request, but even if that was the way it worked, then my JSON is all off!

Looking at the HTTPoison docs, it makes much more sense. That said, how would I send both the file and the JSON in the body? That’s what the API requires, and I think where I’ve been getting confused…

@kokolegorille Thanks for that. I hadn’t seen the standalone wrapper for ImageMagick, but have been using the one that comes bundled with Arc. I wanted to experiment with Kraken purely because I can move the image processing off the app server, and onto some hardware thats kitted out specifically for it. Part premature optimisation, part curiosity. :wink:

Right, that’s for the nudge in the right direction. I’ll make some coffee and see if I can get my head around submitting the file and JSON together.

5

Is there anyway to show what body is being sent? I’ve managed to get past all the Elixir errors, and am now getting a response from the API saying that the incoming request doesn't contain a valid JSON doc.

From what I can see of the (limited) API documentation, it throws this error not because of invalid JSON, but invalid JSON shape for the API.

6

Not sure if HTTPoison or the underlying hackney do have built in functions/capabilities for that, but at a last effort you can use tcpdump/wireshark to analyze sent data.

7

The %HTTPoison.Response struct that you get back from post should have a body field.

{_, resp} = HTTPoison.post(...)
resp.body

Or

HTTPoison.post!(...).body

EDIT: this is the response body though, so that won’t do. What you can do though is make the post request to http://httpbin.org/post and it will echo it back to you in the response so you can check.

8

Just had a crack at using Wireshark, and I’m not ashamed to say it went straight over my head. Managed to follow the TCP stream and filter out just my comms to the API, but in terms of ascertaining what I sent…not a clue.

I’m not sure how I’ve never heard of httpbin, and that gives me some insight into the request, although a file being a part of it makes it very large and very messy. Beyond that, there’s a lot of character escaping; perhaps too much!

I might have to admit defeat on this one…

Going back to the root of my problem, not including using ImageMagick on the server, how are you guys/do you deal with image optimisation on upload (or download)? The site will have a fair amount of user submitted images, and other than Imgix (on the download which is £££) or ImageMagick on the upload, what are my options to minimise what I store and what the website delivers?

EDIT: Sorry, I realise I’m all questions today!

9

The list of fields you pass in inside of the body will be the multi-part form fields. Hackney is expecting a list of key-value tuples:

{:multipart, [{:file, "foo.jpg"}, {"cat", "meow"}, {"dog", "bark"}]}

That will send the file (hackney handles that :file option specially) and set cat=meow&dog=bark for the multi-part form data. It looks like the kraken api is expecting a single form data field to be set (data) that contains a json encoded string. At least that is my assumption based on your curl command. So you will likely need encode the json first and then use that encoded string as a single form field in your request, something like:

json_body = Poison.encode! %{
  "auth" => %{
    "api_key" => "API_KEY",
    "api_secret" => "API_SECRET"
  },
  "wait" => true,
  "dev" =>true
}

req_body = {:multipart, [{:file, "foo.jpg"}, {"data", json_body}]}
HTTPoison.post!("https://api.kraken.io/v1/upload", req_body)
2 Likes
10

Thanks for that. I had been working long the same lines, but hadn’t been encoding the JSON body first. Unfortunately, that still isn’t working, reporting the same “invalid JSON” as before.

I’ve done some more debugging however, and by sending just a 1x1 PNG, it allows me to see the full response from httpbin. Through some trial and error, based on your snippet above, I’ve ended up here:

image = Path.expand("~/example.png")

json = Poison.encode!(%{
  "auth" => %{
    "api_key" => "cdbeb439e39ae537a2b17d3c174093", #3
    "api_secret" => "007486b58928c53ab6d5be5a897a892fa3f03d" #d
  },
  "wait" => true,
  "dev" => true
})

IO.puts(json)

body = HTTPoison.post!("https://httpbin.org/post", {:multipart, [{:file, image, {"form-data", [{"name", "upload"}]}, []}, {"data", json}]}).body

Poison.decode!(body)

Without the “form-data” section on the :file it was sending the file outside of the form scope. Adding that moved it inside, as a sibling of data (see below).

The JSON from the IO.puts comes back as:

{"wait":true,"dev":true,"auth":{"api_secret":"007486b580928ab6d55bea897a892fa3f03d","api_key":"cdbeb439ae537a21bd3c17e4093"}}

Yet my response (decoded) from httpbin is looking like this:

%{"args" => %{}, "data" => "", "files" => %{},
  "form" => %{"data" => "{\"wait\":true,\"dev\":true,\"auth\":{\"api_secret\":\"007486b58092853ab6d5597a892fa3f03d\",\"api_key\":\"cdbeb439e39aea21b3c17e4093\"}}",
    "upload" => <<239, 191, 189, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 13, 73,
      72, 68, 82, 0, 0, 0, 1, 0, 0, 0, 1, 1, 3, 0, 0, 0, 37, 239, 191, 189, 86,
      239, 191, 189, 0, 0, 0, 3, 80, 76, ...>>},
  "headers" => %{"Connection" => "close", "Content-Length" => "576",
    "Content-Type" => "multipart/form-data; boundary=---------------------------grfpskvddsudekct",
    "Host" => "httpbin.org", "User-Agent" => "hackney/1.9.0"}, "json" => nil,
  "origin" => "217.46.86.138", "url" => "https://httpbin.org/post"}

Is the string escaping within the request the last hurdle now, or should that be OK?

NB: I’ve edited the API keys randomly on all these examples… They won’t work

Thanks for all your help with this!

11

Right OK, I’ve compared the working curl commands request using httpbin to what I’ve been sending through HTTPoison.

Here’s the working curl request:

{
  "args": {},
  "data": "",
  "files": {
    "upload": "data:application/octet-stream;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABAQMAAAAl21bKAAAAA1BMVEX/TQBcNTh/AAAAAXRSTlPM0jRW/QAAAApJREFUeJxjYgAAAAYAAzY3fKgAAAAASUVORK5CYII="
  },
  "form": {
    "data": "{\"auth\": {\"api_key\": \"API_KEY\", \"api_secret\": \"API_SECRET\"}, \"wait\":true, \"dev\":true}"
  },
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Content-Length": "478",
    "Content-Type": "multipart/form-data; boundary=------------------------4026e63713e1d44d",
    "Expect": "100-continue",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  "json": null,
  "origin": "217.46.86.138",
  "url": "https://httpbin.org/post"
}

And here’s the non-working HTTPoison one (decoded using Poison.decode!):

%{"args" => %{}, "data" => "", "files" => %{},
  "form" => %{"data" => "{\"wait\":true,\"dev\":true,\"auth\":{\"api_secret\":\"007486b5808c53ab6d5e5a897a892fa3f03d\",\"api_key\":\"cdbeb439ae5371b17d3c17e4093\"}}",
    "upload" => <<239, 191, 189, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 13, 73,
      72, 68, 82, 0, 0, 0, 1, 0, 0, 0, 1, 1, 3, 0, 0, 0, 37, 239, 191, 189, 86,
      239, 191, 189, 0, 0, 0, 3, 80, 76, ...>>},
  "headers" => %{"Connection" => "close", "Content-Length" => "572",
    "Content-Type" => "multipart/form-data; boundary=---------------------------bvwzzrgqeocxmiri",
    "Host" => "httpbin.org", "User-Agent" => "hackney/1.9.0"}, "json" => nil,
  "origin" => "217.46.86.138", "url" => "https://httpbin.org/post"}

The named upload needs to be inside the files part of the request, not the form as I thought in my last post. Now I just need to work out how to name the file within files using Hackney.

Getting there…

12

The file appears inside of the files part for me with the following:

data =
  %{"auth" => %{
    "api_key" => "API_KEY",
    "api_secret" => "API_SECRET"},
  "wait" => true,
  "dev" =>true}

file_headers =
  {"form-data", [
    {"name", "upload"},
    {"filename", "changed_name.jpg"}]}

req_body =
  {:multipart, [
    {:file, "priv/asdf.txt", file_headers, []},
    {"data", Poison.encode! data}]}

Poison.decode! HTTPoison.post!("http://httpbin.org/post", req_body).body

I end up with the following response (“filestuff\n” is the contents of the file I’m uploading):

%{"args" => %{}, "data" => "", "files" => %{"upload" => "filestuff\n"},
  "form" => %{"data" => "{\"wait\":true,\"dev\":true,\"auth\":{\"api_secret\":\"API_SECRET\",\"api_key\":\"API_KEY\"}}"},
  "headers" => %{"Connection" => "close", "Content-Length" => "463",
    "Content-Type" => "multipart/form-data; boundary=---------------------------wklvlhspeptdrocq",
    "Host" => "httpbin.org", "User-Agent" => "hackney/1.9.0"}, "json" => nil,
  "origin" => "76.17.103.160", "url" => "http://httpbin.org/post"}

What weird though is I don’t really see the difference between your hackney request body from the previous post and mine, except for the filename header, but that doesn’t appear to be doing anything.

EDIT: Hah, it looks like having the filename key makes the difference after all. Removing it will move the file out of the files section and into the form section. I don’t see where the file name gets set in the response though… there is always the possibility that httpbin doesn’t echo that info.

13

I just found the exact same thing RE: the filename. Weird how it’s absence changes the request completely!

I’ve now got my curl request to exactly match the HTTPoison one, as far as I can tell from httpbin at least; and it’s still not working. :confounded:

Note to all API creators, sometimes a more specific error message is very helpful! :laughing: Having spent almost two days trying to guess what’s wrong with my request, I’ve come to realise how important proper error messages are!

Thanks so much for your help Kyle! If nothing else I’ve learnt a lot about multipart forms, a little bit of Erlang, and a whole lot about API design!

1 Like
14

Here is something even stranger. I used wireshark to inspect the actual http request being sent to httpbin for both CURL (using what kraken shows in the example) and HTTPoison/Hackney (using what we have from before). The bodies are similiar (fake api keys of course):

HTTPoison

POST /post HTTP/1.1
Host: httpbin.org
User-Agent: hackney/1.9.0
Content-Type: multipart/form-data; boundary=---------------------------osvihhqlhxcgbigd
Content-Length: 317

-----------------------------osvihhqlhxcgbigd
content-length: 111
content-type: application/octet-stream
content-disposition: form-data; name="data"

{"auth":{"api_secret":"5b9312b2901723c11d3351537afdfea1114133ca","api_key":"32b7153129696bad4b8d46275bb3dbb55"}}
-----------------------------osvihhqlhxcgbigd--

CURL

POST /post HTTP/1.1
Host: httpbin.org
User-Agent: curl/7.55.1
Accept: */*
Content-Length: 253
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------538b0b27aa2c25b0

--------------------------538b0b27aa2c25b0
Content-Disposition: form-data; name="data"

{"auth":{"api_key": "e2b153429696ead468d465754b3dfb55", "api_secret": "5b9812b2901713c11d339d537afdfe6111f133ca"}}
--------------------------538b0b27aa2c25b0--

Now if I change only the urls on these requests fromhttp://httpbin.com to https://api.kraken.io/user_status, the curl one will get a successful response from kraken but the HTTPoison one gets the api error (Incoming request body does not contain a valid json object). I’m testing only on the user_status endpoint, so the file upload doesn’t even come into play here.

I’m not really sure what this means yet. I’m wondering if the content-type header for the multi-part that hackney adds is throwing kraken off somehow?

15

Wow, great work! That certainly tallies with the experience I was getting using just the HTTPbin response as my basis for comparison. I guess as you say, there must be something throwing Kraken off.

The only difference I see in the headers (in both your examples, and mine earlier) is the Expect: 100-continue. Perhaps it’s related to this issue? https://github.com/edgurgel/httpoison/issues/124

I had hoped to make this into a package for the community to use, not that I think there’d be a huge demand for it! I think given the problems we’ve run into, it might be something of a non-starter. It certainly seems a very fussy API, and when there’s alternatives such as ImageMagick (either used with Arc or Mogrify) then perhaps we call it a day/weekend? :sunglasses:

16

Okay, I got it working. It did turn out to be the extra multi-part headers that hackney was adding to the data part of the request (the ones it adds the file part seem to be working). I’m not sure if hackney is adding stuff that it shouldn’t or if the kraken api is choking when it shouldn’t. The way I got around it was to manually override the content-type header and set it to ‘text/plain’.

To get it working you need your data to look like this:

file_disposition =
  {"form-data", [
    {"name", "upload"},
    {"filename", "changed_name.jpg"}]}

file_headers = []

data_headers = [
  {"content-type", "text/plain"}]

req_body =
  {:multipart, [
    {:file, @filename, file_disposition, file_headers},
    {"data", json_data, data_headers}]}

HTTPoison.post!(@upload_url, req_body)

Using this I was able to upload an image to kraken without any errors.

I’ve included my full test code below in case that doesn’t work. I found out about another site similar to httpbin that keeps a log of the requests you send it. It also shows the full request payloads, which is nice for troubleshooting things like this: https://requestb.in

I was helping out with an issue earlier this week dealing with ssl in HTTPoison, and in a way it’s similar to this one in that the issue arose from interoping with the underlying hackney/erlang layer. It’s really nice that we have elixir libraries built on top of these battle-tested erlang libraries that are fully-featured, stable, and have a track record of production use, but sometimes it can be jarring whenever you have to cross that erlang/elixir boundary.

I don’t know what the best way to address that is. It could be putting more elixir on top of them, maybe something like HTTPoison.post_multipart() in this case? Or libraries could thoroughly document their erlang layers as if they are parts of the elixir library themselves, rather than ceding to the erlang docs (which while thorough are often not quite as nice as the elixir ones).

Either way, I’m considering creating an HTTPoison issue, I’m not quite sure if this classifies or not though

Full code:

defmodule HTTPoisonJsonFormPost do

  @image_file "priv/5x5.png"

  #@status_url "http://httpbin.org/post"
  #@upload_url "http://httpbin.org/post"

  #@status_url "https://requestb.in/1iuat661"
  #@upload_url "https://requestb.in/1iuat661"

  @status_url "https://api.kraken.io/user_status"
  @upload_url "https://api.kraken.io/v1/upload"

  @api_key "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  @api_secret "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

  def auth(),
    do: %{"api_key" => @api_key, "api_secret" => @api_secret}

  def params_with_auth(params \\ %{}),
    do: %{"auth" => auth()} |> Map.merge(params)

  def format_data_multipart(payload) do
    # overwrite headers for the json data field
    headers = [{"content-type", "text/plain"}]
    {"data", Poison.encode!(payload), headers}
  end

  def do_status() do
    data_multipart = params_with_auth() |> format_data_multipart()
    req_body = {:multipart, [data_multipart]}
    HTTPoison.post!(@status_url, req_body)
  end

  def do_upload() do
    disposition =
      {"form-data", [
        {"name", "upload"},
        {"filename", "changed_name.jpg"}]}

    data_multipart =
      %{"wait" => true, "dev" => true}
      |> params_with_auth()
      |> format_data_multipart()

    # don't overwrite any headers for the file upload
    headers = []

    req_body =
      {:multipart, [
        {:file, @image_file, disposition, headers},
        data_multipart]}

    HTTPoison.post!(@upload_url, req_body)
  end
end
2 Likes
17

Wow…amazing! Thank you so much for obviously spending quite some time on this! I can confirm that this gets it working my end too!

I’m still pretty “green” with Elixir, and before this, had never even looked at Erlang! I’ll agree it’s jarring moving from one to the other, but I put that down to my complete ignore of Erlang, and relative ignorance of Elixir! :laughing:

It’s hard to say, but I’m more inclined to say that it’s the Kraken API choking than it is Hackney adding too much — although it’s clear from reading through their issues and docs whilst trying to solve this that they’ve got a lot of edge cases that they handle, and perhaps because of that it is very explicit.

Either way, thank you so much for your help!

18

Kyle your code was a big help, thanks much. But my gawd, what a PITA. The final change that made it work for me was putting extra quotes on the strings in the disposition section as…

disposition =
{“form-data”, [
{“name”, ~s|“recording”|},
{“filename”, ~s|“newfilename.wav”|}]}

1 Like