Web scraping tools

@_russellb: My list:

1. Floki - fetching data from HTML pages
2. HTTPoison - fetching files (also HTML pages)
3. Poison - decoding JSON for example when you have json in element attribute value
4. NimbleCSV - some data you can fetch from CSV files on some pages for example export search results to CSV
5. ExVCR - recording library for HTTPoison - useful in TDD (Test Driver Development)
6. Retry - just to retry
7. Hound - fallback for HTTPoison and Floki for pages using JavaScript frameworks/libraries

There is also Meeseeks (really good replacement for Floki), but for me it requires Rust that in my case I need to compile (Funtoo Linux) that takes lots of time on my old laptop.

Web scraping is simple

I’d say this is a very generalized statement that should be cleared up for future readers bumping into your post. Not being disrespectful, but saying something like that without some real meat could get a new person in trouble.

But trust me - If you find yourself scraping 5-10 million jobs a day, it quickly becomes “not simple”. The premise of crawling/scraping is not complex, but I can assure you that sustaining it for 7-8 years on end and returning data in a timely fashion to paying customers is not easy at all.

Some more tips from my view, having done this for so long (without a single legal issue):

• Try to form a personal relationship with an IP provider - yes you can use the publicly available providers you mention above and there are tons, but none of those will scale to the numbers I needed to hit in any reasonably economic way. Easier said than done, but ask around - exhaust friends in SEO and marketing fields.
• Start slow. If you only have like 100 IPs to work with, don’t touch a target more than 1 time per hour with the same IP to start unless you plan to treat it like a “smash and grab”.
• Link your IPs to user agents somehow. Meaning if you pull IP #1 and go off to hit a site and randomly grab a UA string to roll with it - make sure the next time you show that IP - you come with the same UA string.
• Have some controls for measuring data quality over time. This may need to be manual in your case. You’d be amazed at how many big sites now start throwing you trash data that looks correct at a glance.
• When you think you have a crawl script ironed out - be sure you toss it at a tool like the EFF Panopticlick (and others) to look for obvious fingerprinting you may have not plugged. https://panopticlick.eff.org/
• Make sure, if using a headless browser you’re plugging all the massive, truck-sized holes they all expose… Things like mocking the navigator.platform (and it better match the UA string) are major gaps I see all the time.

I think the choice depends on whether you need to evaluate js or not.

If you do, then in elixir there is hound [0] for this. I haven’t used it though since I usually use selenium with python. Actually, I usually try to avoid evaluating js completely by figuring out what apis it calls and calling them myself.

If you don’t, then just an html parser like floki [1] will do. You might be interested in reading the source code for magnetissimo [2] to see how a scraper works with it.

[0] GitHub - HashNuke/hound: Elixir library for writing integration tests and browser automation · GitHub
[1] GitHub - philss/floki: Floki is a simple HTML parser that enables search for nodes using CSS selectors. · GitHub
[2] GitHub - sergiotapia/magnetissimo: Web application that indexes all popular torrent sites, and saves it to the local database. · GitHub