Very neat.

Do you respect (or provide the option to respect) robots.txt?

Also, can you compare Crawler to Crawlie?

I only just discovered Crawlie a few days ago, but from a quick glance at the implementations, I think the main differences are 1) scope 2) flow control.

In terms of scope, Crawler offers save to disk / offline, rate limiting, and a bunch of hooks to swap in your own logic. I noticed Crawlie leverages erlang’s pqueue library to provide priority queueing, whereas Crawler simply uses erlang’s builtin FIFO queue.

In terms of flow control, Crawlie uses Flow, whereas Crawler uses its own GenStage implementation that offers worker pooling and rate limiting. I believe the latter offers a bit more fine grained control, especially useful to crawl sources that might have rate limit on the server end.

These are just quick observations, so please correct me if I’m wrong.

I’d love people to try both and give both projects some love and feedback.

Wow, it’s been… how many years? I have just picked up Crawler again after a looong hiatus. Just released v1.2.0 with memory usage improvement.