oltarasenko
Crawly - A high-level web crawling & scraping framework for Elixir
Dear Elixir community,
After a year of development, bug fixes, and improvements, we are proudly ready to share the release of Crawly 0.10.0 here with you.
Check out the source code here: GitHub - elixir-crawly/crawly: Crawly, a high-level web crawling & scraping framework for Elixir. · GitHub
Check out our experimental, pre-alpha visual UI here: http://18.216.221.122/ and play with jobs scheduling there.
We have dedicated a lot of time and knowledge to build a fast and feature-rich web scraping framework. To be absolutely honest, I have to say that I took a lot of ideas from other popular web scraping framework (Scrapy, python), as I have previously worked with the Scrapy core team.
We have some reported production usages of Crawly (some of them with really long-running crawls), however, we still have to approach the stable version (hopefully it will happen in a couple of releases).
To describe how Crawly is different from other known elixir scraping frameworks, I will list crawly’s features which I believe make it outstanding:
- Documentation - we have spent an enormous amount of time and effort to build great and clear versioned documentation!
- Rate limiting
- Robots.txt support
- Requests and Items validators
- Automatic duplications filtering
- Automatic cookies management (allows to bypass login pages and cookie-based regional filtering)
- Browser rendering (with the help of Splash)
- Retries support
- Proxies support
- HTTP API
- Visual jobs management dashboard which allows operating multiple Crawly nodes at the same time (experimental): see it deployed on demo ec2 micro instance: http://18.216.221.122/
We hope it will be useful for you!
If you have a suggestion or a production use case you’re happy for us to share, please get in touch.
Most Liked
oltarasenko
Hey people,
I’ve got a bit of time to work on Crawly recently, and as a result have made a new release and a new article about it 
Hopefully you will find it useful.
oltarasenko
As a part of the project development, I have decided to create a short cookbook of scraping recipes, if you’re using Crawly or Scraping in general, these articles might be useful for you (I am including medium friend links, so everyone can read them):
https://oltarasenko.medium.com/web-scraping-with-elixir-and-crawly-browser-rendering-afcaacf954e8?sk=345c1777d793aa99828986010cc3b077
oltarasenko
I can also refer my talk from November 2019 explaining how scraping can be used in general: https://www.youtube.com/watch?v=ovSQGlkakAQ
Popular in Announcing
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Courses that'll move you from confusion to "Aha, now I get it!"