I can’t tell you exactly what you need to include, but here is a good approach to find out what you need:

1. Make a manual request via your browser and record it using the devtools. Make sure you are in incognito mode so you don’t have any session data.

2. Recreate the request, including all the headers and parameters using a HTTP tool, e.g. Postman.

3. If step 2 works, then start to remove some headers which you think are not needed. The goal is to end up with just the stuff you need.

4. Recreate the request using HTTPoison. It’s mostly just headers and in some cases you might also need some cookies.

The approach is pretty simple. It’s basically just emulating a real person. Have a look at headers like User-Agent, Accept, etc. Some sites try to be fancy and use those to detect crawlers.

If, for some reason, you need a cookie. Then figure out where and how the cookie is set. This means, you might end up with an extra request to obtain a cookie which you then use for the following requests.

Or use the cURL command to import into Postman Fiddling around with a cURL command that contains headers and cookies is pretty awful

Just to help you with some future problems, there are more that you probably don’t expect. Let me list them below:

1. This goes against Google’s TOS. You can be in legally not great position if you do that. I’m not saying Google will actually come after you but you have to be aware that this is not valid usage of the service, from Google’s point of view.

2. They not only say this is a violation and not do anything about it, but they will actively detect and block you from scraping search results. Google builds profiles of all browsers / users with cookies and also IP addresses, and it will record these, do some matching behind the scenes and figure out that there is a suspicious browser, coming from a suspicious IP address, and will do several things: first, they will present you with a captcha to make sure “you are not a robot”. They can also block you even if you keep solving the captcha, effectively blocking off the IP address from Google’s services. You will have to rotate the IP addresses on regular basis and build browsers’ “human-like” behavior pattern to prevent being detected. You probably want to slow things down, also visit other sites, make your profile seem like a real user etc.

This is still doable, esp. if you want to do it at small scale for own usage, but at large scale it becomes expensive, risky, and you may not want to do it.