Does floki with either html5ever or mochiweb_html normalize the HTML?
I’ve used nokogiri on the Ruby side and one of my problems with it is that it normalizes the HTML.
veverkap
mischov
@veverkap
I am not exactly sure what you mean by normalizing the HTML, but both html5ever and mochiweb_html attempt to interpret malformed HTML into valid HTML, which means that the result of parsing might not be exactly the HTML you passed in (but generally in a good way).
If that doesn’t answer your question, could you give a more concrete example?
1
veverkap
That’s exactly what I mean.
We have a tool that we use to scrape HTML pages. When we tried to use Mechanize (which uses Nokogiri under the hood), it would correct the HTML, which sometimes gave different results.
You answered my question, thanks!
0
sntran
I also started a little scraping framework based on the Python-based Scrapy. It’s on GitHub - sntran/scrapex: An Elixir open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way. Or just an experiment writing a scraper in Elixir to scratch my own itch. Use at your own risk. · GitHub
It uses HTTPoison for making HTTP request, and Floki to parse HTML. I tried not to bring the whole headless browser in, so it does not handle SPA.
2
adammokan
@_russellb - I’ve been using a combo of NightmareJS via ports (using Porcelain). I’ve been scraping at a pretty large scale since last summer with Elixir. There are some more details in some previous threads of mine if you search.
The only downfall with Nightmare or any JS-capable solution is the CPU/memory resources needed. If you are just needing HTML capabilities, HTTPoison (or swap with HTTPotion, raw hackney, etc) works fine in my experience. Unfortunately, I do not have the HTML only luxury as I’m dealing with single page app, JS required systems. As far as the heavy resource load for JS stuff via headless browsing - I have a distributed elixir/erlang solution in place with dedicated scraping nodes I can bring up dynamically and dispatch work to from a master node. Everything is running headless on linux boxes (some in our own datacenter and some on AWS spot instances).
For parsing the results, I use Floki like most people do. Speed is never an issue here for me so I’ve not bothered with the other parsers out there.
FWIW, we came from a lot of legacy Ruby/Mechanize code. The Elixir stuff is much, much easier to maintain and way more stable. I do have times where Nightmare acts up and doesn’t release resources, but I resolve that with some elixir-based cleanup tasks that run every few minutes to check for zombie processes that get missed. Porcelain helps a lot for the port logic, but is not 100%. I also keep all crawling off my main logic node so I can afford to lose a crawl and just bounce a node without sweating it. I’d recommend something similar if you need large scale throughput. I just assume the job was lost if I do not get a reply back from a crawling node within five minutes and send it to another node.
Good luck!
5
bzhr
I’ve been using Scrapy for a while and it is a nice and mature framework. I am writing Xpath selectors for getting data from htmls and using the built in Scrapy system for following links and building pipelines. There are some things that can be improved in Scrapy though. You need to write your own spider management process, when you have many spiders in a projects this gets complex. Also, today the same url that is crawled can contain frequently updated data. Scrapy is designed to through a bunch of URLs and store the data somewhere, but today the web works differently, I need to know if a field on a web site is updated. Seems like things like these can be managed very well in Elixir. Using xpath selectors (or the mentioned Rust implementation) is easy. Some html diff logic on top of that in order to listen for updates would be nice to have instead of how Scrapy functions.
1
bzhr
oltarasenko
I wanted to add this tiny library: GitHub - elixir-crawly/crawly: Crawly, a high-level web crawling & scraping framework for Elixir. · GitHub to our discussion list. You might find it useful, for the web scraping case.
I think it’s quite well documented, and could be used as a good starting point. Please find the relevant tutorial here:
3
jihantoro
Web scraping is simple, IMHO there is two type of site :
- Server-Side Rendered page
easy to scrape with curl or any HTTP Client (+ HTML Parser) - Javascript Rendered page
A. You can use Headless / Browser Automation, but, it will be slow (+ HTML Parser)
B. Do “little bit” Reverse Engineering on their Web API (FASTER)
Important Point :
- Make sure your scrapper support Proxy Usage
- If your site target has anti-scraper / crawler / bot (Like your bot follow pagination, 1->2->3 and so on) and it block your IP, you can use IP Rotation Service like geosurf.com and luminati.io
- In some Country / Site, Web Scraping are prohibited
4
oltarasenko
Thanks for your comment.
I would not say that web scraping is simple. Everything depends on the scale. Probably for very small sites you can use curl or wget. But I am talking about cases when you need to scrape millions of pages. And in these cases, things are way more complex, as you have to solve concurrency and resource management problems. Also, finding a good strategy of crawling a million pages is a challenge (also consider cases when URLS are dynamically generated)!
To answer your comments:
- Server-Side Rendered page
easy to scrape with curl or any HTTP Client (+ HTML Parser)
Will not scale. Also, you will have to avoid visiting pages twice and filtering out duplicates
- Javascript Rendered page
A. You can use Headless / Browser Automation, but, it will be slow (+ HTML Parser)
In most of the cases, you will be able to find how a web page (e.g. a product page) is fetching data from API, so in most of the cases, you don’t need selenium.
B. Do “little bit” Reverse Engineering on their Web API (FASTER)
Unfortunately, this does not work. Most of the web sites do not have API. And those who have, would not provide a full and up to date data. Even more, some of the APIs are just horrible and can’t be used for data extraction.
Important Point :
- Make sure your scrapper support Proxy Usage
It does
- If your site target has anti-scraper / crawler / bot (Like your bot follow pagination, 1->2->3 and so on) and it block your IP, you can use IP Rotation Service like geosurf.com and luminati.io
You’re right. But please take into that proxies management is a complex stand-alone task. There are some quite advanced systems which allow overcoming bans with proxies, and I was developing one of them in the past.
Also nowadays in some cases, it’s just not enough just to perform a request through another proxy, as the most advanced system would also perform 3-4 levels of request fingerprint analysis. With this regards, I would suggest looking at Crawlera
- In some Country / Site, Web Scraping are prohibited
Well.. is that correct to assume the internet is also prohibited in these countries? Please take into account that no search engine can work without web scraping. And I don’t see the web without search these days. (But it’s just an opinion).
4