Your scraper application are depends on you (the logic, structure and so along) and i just want to tell ways to do scraping
And choose that fit with your site target
jihantoro
adammokan
Web scraping is simple
I’d say this is a very generalized statement that should be cleared up for future readers bumping into your post. Not being disrespectful, but saying something like that without some real meat could get a new person in trouble.
But trust me - If you find yourself scraping 5-10 million jobs a day, it quickly becomes “not simple”. The premise of crawling/scraping is not complex, but I can assure you that sustaining it for 7-8 years on end and returning data in a timely fashion to paying customers is not easy at all.
Some more tips from my view, having done this for so long (without a single legal issue):
- Try to form a personal relationship with an IP provider - yes you can use the publicly available providers you mention above and there are tons, but none of those will scale to the numbers I needed to hit in any reasonably economic way. Easier said than done, but ask around - exhaust friends in SEO and marketing fields.
- Start slow. If you only have like 100 IPs to work with, don’t touch a target more than 1 time per hour with the same IP to start unless you plan to treat it like a “smash and grab”.
- Link your IPs to user agents somehow. Meaning if you pull IP #1 and go off to hit a site and randomly grab a UA string to roll with it - make sure the next time you show that IP - you come with the same UA string.
- Have some controls for measuring data quality over time. This may need to be manual in your case. You’d be amazed at how many big sites now start throwing you trash data that looks correct at a glance.
- When you think you have a crawl script ironed out - be sure you toss it at a tool like the EFF Panopticlick (and others) to look for obvious fingerprinting you may have not plugged. https://panopticlick.eff.org/
- Make sure, if using a headless browser you’re plugging all the massive, truck-sized holes they all expose… Things like mocking the
navigator.platform(and it better match the UA string) are major gaps I see all the time.
13
mythicalprogrammer
They also have splash which is their own html/javascript renderer.
I think it’s straight up professional and industrial.
I’ve seen floki and I’ve done beautifulsoup. Scrapy straight up is the tool you want if you want to webscrape website you can scrape like 90% or more of the websites out there with it.
I do webscrapping professionally for one startup stint and I’ve been doing it on the side for the side projects. I also got paid to webscrape linkedin and it didn’t go anywhere, linkedin is pretty dang hard unless you’re willing to put in lots of man hours into it.
I am willing to put my reputation on that tool. It’s really really good. It’s python though.
2
dimitarvp
Extremely valuable comment, thank you for it!
Can you clarify on that? I’ve heard sites like Amazon deliberately give you wrong prices if they detect a bot, is that true for them and others? Or what kind of trash data?
Additionally, how do you even test your bot against tools like Panopticlick at all? Have your bot GET their root page and click the “Go” button? Is that what you meant, or do they (or others) have a dedicated bot testing toolkit?
1
adammokan
Can you clarify on that? I’ve heard sites like Amazon deliberately give you wrong prices if they detect a bot, is that true for them and others? Or what kind of trash data?
Yes. I don’t want to speak for specific sites - but I can tell you many ‘popular’ sites will start delivering results that are either not ordered correctly (if you think in terms of SERP rank data as an example, where order matters to buyers), E-commerce sites will start throwing bogus prices, and so-on. The only real way to protect/test this is to do manual A/B comparisons. The other thing to consider is we’re fully immersed in a world of personalized content - so absolutely it gets difficult, even if not being thrown bogus data, is that your clients will think your results are wrong because they are viewing personalized content if they manually compare results.
Additionally, how do you even test your bot against tools like Panopticlick at all? Have your bot GET their root page and click the “Go” button? Is that what you meant, or do they (or others) have a dedicated bot testing toolkit?
I should have been more specific on that. I mostly reused the same boilerplate headless crawl scripts and would include the site-specific nav/logic separately. So this meant all of my ‘pre-crawl’ setup to plug gaps like setting a legit navigator.platform and making sure navigator.webdriver returns false, etc (there are quite a few of these you need to cover).
Anyhow, I would traditionally include that initial setup logic and then have a custom script that would navigate the Panopticlick site, run the full checks, screenshot the results, and study later. Just to make sure I wasn’t missing something obvious. So yes, click the go button - load the full results and screenshot the page.
Aside from that there are many other “how private is my browser” checks out there that test for hardware-level info that may be worth mocking on some targets.
This reminds me of the fun I had with geolocation/gps coords - again, depending on what you are going for. Most common case for that effort was a well-known map site. Some hints for geolocation - and this may be outdated, but it was always important to mock both navigator.geolocation.getCurrentPosition as well as navigator.geolocation.watchPosition to spoof lat/long. A little bit of noise in the coords.accuracy attribute went a long way here 
1
_rubenfa
Geolocated results are pretty interesting too. I had a long battle with that in a previous job. You can try to use IPs from the destination country but is not always easy. I remember to obtain interesting results when the IP is geolocated in a country border.
Finally, we made it work by mocking geolocation properties as you said. It was especially useful for geolocated results in regions of the same country.
I think scraping concept is easy to understand, but there is a ton of variables to take in mind, depending on the stuff you are scraping. Some of them make the process really hard to implement.
1
oltarasenko
Hey people. Just wanted to announce that I have made an article on Erlang Solutions blog, about using Crawly: Erlang Solutions
Also I have made a couple of sample/tutorial projects to get started with: GitHub - oltarasenko/crawly-spider-example · GitHub
GitHub - oltarasenko/products-advisor · GitHub
GitHub - oltarasenko/crawly-cars · GitHub
Hopefully they could be interesting for people doing web scraping.
4
mischov
@oltarasenko Couple bits of feedback that are common mistakes I see people making when using Floki.
-
Recommend a safer HTML parser than Floki+mochiweb_html.
I know it’s nice to not need to start an Elixir article with “and then install Rust”, but web scraping is exactly the situation you do want an HTML5 compliant parser because you don’t know how well formed the HTML will be and mochiweb_html (Floki’s default parser) can incorrectly parse parts of the HTML if it’s malformed (and potentially just drop those parts silently).
At the very least use the html5ever parser with Floki, though you may have trouble getting it to compile since html5ever_elixir hasn’t been updated for nine months despite an outstanding need to upgrade Rustler so that it works with more recent versions of Erlang/OTP.
Better yet, use Meeseeks instead of Floki because it will by default provide you an HTML5 compliant parser based on html5ever that does compile on the latest versions of Erlang/OTP.
Floki’s mochiweb_html parser has a place, mainly in situations where you are dealing with known, well-formed HTML and you don’t need the weight of an HTML5 compliant parser (like when you’re testing your Phoenix endpoints), but people should know the risk they’re taking if they use it for web scraping.
-
Stop parsing each page four times.
When you run
response.body |> Floki.find(...), you’re really running the equivalent ofresponse.body |> Floki.parse() |> Floki.find(...)which means your fourFloki.finds are parsing the whole document four times.Instead, try
parsed_body = Floki.parse(response.body)thenparsed_body |> Floki.find(...). -
Don’t select over the whole document when you don’t need to.
Three of your selectors are:
"article.blog_post h1:first-child","article.blog_post p.subheading"and"article.blog_post". That means you’re selecting the samearticle.blog_postthree times, then making sub-selections two of those times. Instead, try something like:parsed_body = Floki.parse(response.body) blog_post = Floki.find(parse_body, "article.blog_post") title = blog_post |> Floki.find("h1:first_child") |> Floki.text author = blog_post |> Floki.find("p.subheading") ...Doing that means that instead of walking the whole document each time you want to make a sub-selection you just walk the the portion you’re interested in. In this case when there is only one of the thing you’re making sub-selections on it’s probably not a huge difference, but in cases where you’re sub-selecting over a list of items it can add up.
6
mischov
Here is a version of your crawler that uses Meeseeks and fixes all of the above problems.
defmodule Esl do
@behaviour Crawly.Spider
import Meeseeks.CSS
@impl Crawly.Spider
def base_url() do
"https://www.erlang-solutions.com"
end
@impl Crawly.Spider
def init() do
[
start_urls: ["https://www.erlang-solutions.com/blog.html"]
]
end
@impl Crawly.Spider
def parse_item(response) do
# Parse the response body as html, though Meeseeks comes with an
# XML parser too if you want to parse the blog posts from the
# RSS feed instead
parsed_body = Meeseeks.parse(response.body, :html)
# Get new urls to follow
urls =
parsed_body
|> Meeseeks.all(css("a.more"))
|> Enum.map(&Meeseeks.attr(&1, "href"))
# Convert urls into requests
requests =
Enum.map(urls, fn url ->
url
|> build_absolute_url(response.request_url)
|> Crawly.Utils.request_from_url()
end)
# Extract item from a page, e.g.
# https://www.erlang-solutions.com/blog/introducing-telemetry.html
#
# Find the post using `Meeseeks.one`, which will stop after
# the first match rather than looking through the rest of the
# document for something it will never find more of.
post = Meeseeks.one(parsed_body, css("article.blog_post"))
title =
post
|> Meeseeks.one(css("h1:first-child"))
|> Meeseeks.own_text()
# `Meeseeks.own_text` trims automatically and only gets the text
# from the selected element.
author =
post
|> Meeseeks.one(css("p.subheading"))
|> Meeseeks.own_text()
# `Meeseeks.text` will get the combined text from the element and
# all of its descendants
text = Meeseeks.text(post)
%Crawly.ParsedItem{
:requests => requests,
:items => [
%{
title: title,
author: author,
text: text,
url: response.request_url
}
]
}
end
def build_absolute_url(url, request_url) do
URI.merge(request_url, url) |> to_string()
end
end
3
oltarasenko
Hey @mischov!
Thanks for the feedback. Actually I was aware that parsing would happen 3 times, but decided not to do anything with it for now. At this point, I think that it’s better to update examples, so they are not misleading! Thanks for the hint.
I don’t have problems with Rust. Parsing should be done by low level and fast languages! Basically, scrapy uses C parses in order to parse pages under the hood. We should do the same.
Q: I am currently looking for the XPath library, to use it in Crawly. But at this point, nothing works properly when in comes to HTML pages. Are you aware of the options to try XPath selectors?
Again huge thanks for comments!
1