Crawler - easy web crawling / scraping powered by GenStage

I will try to use it to crawl data

Very neat.

Do you respect (or provide the option to respect) robots.txt?

Also, can you compare Crawler to Crawlie?

I only just discovered Crawlie a few days ago, but from a quick glance at the implementations, I think the main differences are 1) scope 2) flow control.

In terms of scope, Crawler offers save to disk / offline, rate limiting, and a bunch of hooks to swap in your own logic. I noticed Crawlie leverages erlang’s pqueue library to provide priority queueing, whereas Crawler simply uses erlang’s builtin FIFO queue.

In terms of flow control, Crawlie uses Flow, whereas Crawler uses its own GenStage implementation that offers worker pooling and rate limiting. I believe the latter offers a bit more fine grained control, especially useful to crawl sources that might have rate limit on the server end.

These are just quick observations, so please correct me if I’m wrong.

I’d love people to try both and give both projects some love and feedback.

Hi guys and @fredwu . I`m trying to start crawler. I have a trouble with setting filter parameter. Can you please give me a clue? Thanks.

What sort of issues are you having?

I have a module:

defmodule CrawlFilter do
  @behaviour Crawler.Fetcher.UrlFilter.Spec

  def filter(Preformatted texturl, opt \\ []) do
    String.match?(url, ~r/example\.com/)
  end
end

It’s a harcoded filter for test
I call Crawler like:

C> rawler.crawl(“http://example.com”, [save_to: “/path/to/save”, url_filter: CrawlFilter])

In this case error is rise (MatchError) no match of right hand side value: true

I’m new to Elixir and sorry if my question is stupid.

Damn, I need to return tuple instead of boolean. It’s fixed.

I’m glad it’s worked out for you.

This is very useful library. Congrats for your work!

Thank you, @fredwu, for creating this library and for writing clear documentation. The high-level architecture diagram was particularly useful for a beginner like me to get a good overview.

I’m planning on trying out your library for my first real Elixir project.

One thing I’m wondering about is whether it would be feasible to assign different IP addresses to each crawler via some kind of proxy or VPN service, such as Tor, TorGuard or NordVPN.

When scraping, I want to respect each site’s robots.txt. I’m also thinking about using SchedEx to trigger scraping at night-time (low-traffic hours), to be mindful of the target sites’ performance.