HTTP POST Requests with nested JSON form-data

Leo328

Hey there,

I’m pretty new to the Elixir world but love what I see so far - you guys and gals seem to be really helpful and supportive people.

I’ll be upfront, I’m trying to scrape data and of course this starts with the basics of trying to mimic HTTP Requests which you see in a browser.

I’m hoping I can stick in this world and am fighting hard to avoid using Ruby or Python for my early stage in my project as I want to get better with Elixir. The Python community is really advanced in this area.

So I go into Chrome DevTools, find the relevant network request and copy the Curl and was hoping to find an easy way to just pass this into HTTPoison or similar library - I haven’t been able to find an easy solution to this.

In Python world, I found this super helpful tool:
https://curl.trillworks.com/ - I was able to simply copy/paste the curl request in, ran the request and got the expected response. It just works.

The form data I’m trying to work with sends JSON which is deeply nested.

Q1. Does HTTPoison or any similar library support this? I haven’t found any documentation for HTTPoison for this use-case.

Q2. How I can I easily convert these requests from DevTools and use this in Elixir? Its not a good workflow to have to manually convert all the similar requests from curl to an Elixir List with tuple pairs.

JSON example:

{"partData":{"groupindex":"414","listing_data_essential":{"groupindex":"414","carcode":"1411579","parttype":"1684","partkey":"10898080","opts":{"0-0-8-1":{"warehouse":"77478","whpartnum":"NAD D946:P","optionlist":"0","paramcode":"0","notekey":"8","multiple":"1"}}},"listing_data_supplemental":{"partnumber":"D946","catalogname":"NAD XL","belongstolisting":"413","markets":["US","CA"],"sortgroup":0,"sortgrouptext":"Economy","showhide":{}},"OptKey":"0-0-8-1"}}

Raw format:

%7B%22partData%22%3A%7B%22groupindex%22%3A%22414%22%2C%22listing_data_essential%22%3A%7B%22groupindex%22%3A%22414%22%2C%22carcode%22%3A%221411579%22%2C%22parttype%22%3A%221684%22%2C%22partkey%22%3A%2210898080%22%2C%22opts%22%3A%7B%220-0-8-1%22%3A%7B%22warehouse%22%3A%2277478%22%2C%22whpartnum%22%3A%22NAD%20D946%3AP%22%2C%22optionlist%22%3A%220%22%2C%22paramcode%22%3A%220%22%2C%22notekey%22%3A%228%22%2C%22multiple%22%3A%221%22%7D%7D%7D%2C%22listing_data_supplemental%22%3A%7B%22partnumber%22%3A%22D946%22%2C%22catalogname%22%3A%22NAD%20XL%22%2C%22belongstolisting%22%3A%22413%22%2C%22markets%22%3A%5B%22US%22%2C%22CA%22%5D%2C%22sortgroup%22%3A0%2C%22sortgrouptext%22%3A%22Economy%22%2C%22showhide%22%3A%7B%7D%7D%2C%22OptKey%22%3A%220-0-8-1%22%7D%7D

Thanks for taking the time to read this. Appreciate any help I can get.

I acknowledge that I may even be going about this completely the wrong way - any tips or suggestions are always helpful.

3 comments

#httpoison #web-scraping

0 3642 3

2020-06-21 03:55:15 UTC

First 3 of 3 Posts

cpgo

The tool you linked has the option to convert the curl request to elixir, is it not working?

Post #1

Leo328

Hi cpgo,

Thanks for your reply - I felt quite silly after you pointed that out. But it doesn’t seem to work in my case:

iex(4)> response = HTTPoison.request(request)
** (ArgumentError) argument error
    :erlang.iolist_to_binary([{"func", "getbuyersguide"}, {"payload", "{\"partData\":{\"groupindex\":\"414\",\"listing_data_essential\":{\"groupindex\":\"414\",\"carcode\":\"1411579\",\"parttype\":\"1684\",\"partkey\":\"10898080\",\"opts\":{\"0-0-8-1\":{\"warehouse\":\"77478\",\"whpartnum\":\"NAD D946:P\",\"optionlist\":\"0\",\"paramcode\":\"0\",\"notekey\":\"8\",\"multiple\":\"1\"}}},\"listing_data_supplemental\":{\"partnumber\":\"D946\",\"catalogname\":\"NAD XL\",\"belongstolisting\":\"413\",\"markets\":[\"US\",\"CA\"],\"sortgroup\":0,\"sortgrouptext\":\"Economy\",\"showhide\":{}},\"OptKey\":\"0-0-8-1\"}}"}, {"api_json_request", "1"}, {"sctchecked", "1"}, {"scbeenloaded", "false"}, {"curCartGroupID", ""}])
    (hackney 1.16.0) /home/leo/wheelknowledge/elixir_scraper/deps/hackney/src/hackney_request.erl:352: :hackney_request.handle_body/4
    (hackney 1.16.0) /home/leo/wheelknowledge/elixir_scraper/deps/hackney/src/hackney_request.erl:87: :hackney_request.perform/2
    (hackney 1.16.0) /home/leo/wheelknowledge/elixir_scraper/deps/hackney/src/hackney.erl:376: :hackney.send_request/2
    (httpoison 1.5.1) lib/httpoison/base.ex:786: HTTPoison.Base.request/6

We’re almost there though.

Post #2

Leo328

Found this hand-rolled function that was useful to converting the maps into keyword lists:

defmodule MapUtility do
  def to_keyword_list(map) do
    Enum.map(map, fn {k,v} ->
      v = cond do
        is_map(v) -> to_keyword_list(v)
        true      -> v
      end

      {String.to_atom("#{k}"), v}
    end)
  end
end

Post #3

— All posts loaded —