oltarasenko
Dear Elixir community,
After a year of development, bug fixes, and improvements, we are proudly ready to share the release of Crawly 0.10.0 here with you.
Check out the source code here: GitHub - elixir-crawly/crawly: Crawly, a high-level web crawling & scraping framework for Elixir. · GitHub
Check out our experimental, pre-alpha visual UI here: http://18.216.221.122/ and play with jobs scheduling there.
We have dedicated a lot of time and knowledge to build a fast and feature-rich web scraping framework. To be absolutely honest, I have to say that I took a lot of ideas from other popular web scraping framework (Scrapy, python), as I have previously worked with the Scrapy core team.
We have some reported production usages of Crawly (some of them with really long-running crawls), however, we still have to approach the stable version (hopefully it will happen in a couple of releases).
To describe how Crawly is different from other known elixir scraping frameworks, I will list crawly’s features which I believe make it outstanding:
- Documentation - we have spent an enormous amount of time and effort to build great and clear versioned documentation!
- Rate limiting
- Robots.txt support
- Requests and Items validators
- Automatic duplications filtering
- Automatic cookies management (allows to bypass login pages and cookie-based regional filtering)
- Browser rendering (with the help of Splash)
- Retries support
- Proxies support
- HTTP API
- Visual jobs management dashboard which allows operating multiple Crawly nodes at the same time (experimental): see it deployed on demo ec2 micro instance: http://18.216.221.122/
We hope it will be useful for you!
If you have a suggestion or a production use case you’re happy for us to share, please get in touch.
Where Next?
Trending in Announcing
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Bringing learning, work and compliance together since 1987.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #elixirconf-eu
- #metaprogramming
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
dgreiss
Congrats and thank you for launching this for the community.
obsidienne
Hi,
Is crawly the right tool to create something like inoreader ?
oltarasenko
If it’s about extracting data from multiple RSS feeds - yes.
oltarasenko
I can also refer my talk from November 2019 explaining how scraping can be used in general: https://www.youtube.com/watch?v=ovSQGlkakAQ
mischov
Congrats on the release.
I still think you should be at the very least suggesting people use Floki’s
html5everparser. Scraping is exactly the context when themochiweb_htmlparser not being HTML5 compliant is most problematic.Brainiac
The use of Floki in the docs is only for demonstrative purposes, since Crawly does not dictate how your data is parsed. Its faster to get up and running without having to install rust, after all.
mischov
Faster, yes, but in my experience people tend towards doing what they see in examples, particularly when the drawbacks of that approach aren’t discussed.
oltarasenko
We’re trying to show different examples. E.g. in one of the most recent articles I was using the Rust based lib called meeseeks.
But indeed, I tend to use Floki in examples, as it’s easier to start this way.
Regarding the comment about drawbacks - I would say the HTML parser is definitely a thing to discuss and highlight. But what we’re also trying to do is to bring the scraping experience to the level of the process: when you can build a spider, schedule a job and validate (QA) results in a visual way.
mischov
Good choice.
Discussing and highlighting parsers is all I was suggesting. Most examples, articles, etc, in the Elixir ecosystem don’t, and so I don’t think people get a chance to learn about potential pitfalls and trade-offs.
It’s great to see the UI and also that you’ve been working hard to be a good citizen by limiting impact and respecting robots.txt. Keep up the good work.
dimitarvp
That’s not that slow or complex either. Going to https://rustup.rs and installing the Rust toolchain has honestly been the most painless tech installing experience I ever had – even on Windows (which can’t be said for the C/C++ toolchain; it’s a nightmare to get several Ruby and Elixir projects to compile their native dependencies without at least one library refusing to compile).
I wish every other language adopted such a model of seamless and “just works” installation that literally takes a minute.