Hey there,

I’m pretty new to the Elixir world but love what I see so far - you guys and gals seem to be really helpful and supportive people.

I’ll be upfront, I’m trying to scrape data and of course this starts with the basics of trying to mimic HTTP Requests which you see in a browser.

I’m hoping I can stick in this world and am fighting hard to avoid using Ruby or Python for my early stage in my project as I want to get better with Elixir. The Python community is really advanced in this area.

So I go into Chrome DevTools, find the relevant network request and copy the Curl and was hoping to find an easy way to just pass this into HTTPoison or similar library - I haven’t been able to find an easy solution to this.

In Python world, I found this super helpful tool:
https://curl.trillworks.com/ - I was able to simply copy/paste the curl request in, ran the request and got the expected response. It just works.

The form data I’m trying to work with sends JSON which is deeply nested.

image993×301 12.1 KB

Q1. Does HTTPoison or any similar library support this? I haven’t found any documentation for HTTPoison for this use-case.

Q2. How I can I easily convert these requests from DevTools and use this in Elixir? Its not a good workflow to have to manually convert all the similar requests from curl to an Elixir List with tuple pairs.

JSON example:

{"partData":{"groupindex":"414","listing_data_essential":{"groupindex":"414","carcode":"1411579","parttype":"1684","partkey":"10898080","opts":{"0-0-8-1":{"warehouse":"77478","whpartnum":"NAD D946:P","optionlist":"0","paramcode":"0","notekey":"8","multiple":"1"}}},"listing_data_supplemental":{"partnumber":"D946","catalogname":"NAD XL","belongstolisting":"413","markets":["US","CA"],"sortgroup":0,"sortgrouptext":"Economy","showhide":{}},"OptKey":"0-0-8-1"}}

Raw format:

%7B%22partData%22%3A%7B%22groupindex%22%3A%22414%22%2C%22listing_data_essential%22%3A%7B%22groupindex%22%3A%22414%22%2C%22carcode%22%3A%221411579%22%2C%22parttype%22%3A%221684%22%2C%22partkey%22%3A%2210898080%22%2C%22opts%22%3A%7B%220-0-8-1%22%3A%7B%22warehouse%22%3A%2277478%22%2C%22whpartnum%22%3A%22NAD%20D946%3AP%22%2C%22optionlist%22%3A%220%22%2C%22paramcode%22%3A%220%22%2C%22notekey%22%3A%228%22%2C%22multiple%22%3A%221%22%7D%7D%7D%2C%22listing_data_supplemental%22%3A%7B%22partnumber%22%3A%22D946%22%2C%22catalogname%22%3A%22NAD%20XL%22%2C%22belongstolisting%22%3A%22413%22%2C%22markets%22%3A%5B%22US%22%2C%22CA%22%5D%2C%22sortgroup%22%3A0%2C%22sortgrouptext%22%3A%22Economy%22%2C%22showhide%22%3A%7B%7D%7D%2C%22OptKey%22%3A%220-0-8-1%22%7D%7D

Thanks for taking the time to read this. Appreciate any help I can get.

I acknowledge that I may even be going about this completely the wrong way - any tips or suggestions are always helpful.