Web scraping tools

This looks amazing, hope you find the time to develop it further!

Hey people, sorry for restoring this a bit old topic again, but I want to add another library called CrawlyUI to the discussion.

You might wonder why I decided to add another library here, as well as there are plenty here already. Well, I believe it’s somehow different as:

1. It simplifies the preview of extracted data
2. Allows to schedule jobs on different nodes
3. Allows defining spiders from the UI, so you don’t have to write code at all, and will get a spider capable of extracting data.

You can find the deployed version here: http://crawlyui.com/
And a short explanation on how to create a spider here: https://www.youtube.com/watch?v=1O-XArob5rU&ab_channel=OlegTarasenko

Really hope you will like it!

Hey, @dogweather I just wonder how it goes? E.g. I wonder if you have tried it out? Did you have any issues?

From our side, I have asked my colleagues to have a glance at the target you have shared. Taking into account that crawly can’t save data in a nested JSON, I have advised them to use a flat structure, injecting all fields into one basic item.

So far our results are far away from being ideal
see: http://crawlyui.com/spider?spider=Elixir.Spiders.OregonRegs

I wonder if you tried it from your side, and if you have managed to gather more items than we did.

Excellent - we have several scrapers running nightly, and are continually writing more, to extract online laws from various countries and locations. My startup, https://public.law, makes the laws easier to read and search.

I use scrapers to create JSON representations of each legal resource. Usually, a collection of laws are hierarchical - a tree of e.g., Divisions > Chapters > Sections. And so I like to write scrapers which return a single JSON tree because it’s easier to capture the parent/child relationships while the spider is running, than to try to recreate it on import.

• A hierarchical Haskell scraper: GitHub - public-law/nevada-revised-statutes-parser: Parses the Nevada NRS into well formed JSON · GitHub
• A hierarchical Scrapy scraper: GitHub - public-law/open-gov-crawlers: Parse government documents into well formed JSON · GitHub

The final result, after import, is a website like this: Nevada Revised Statutes (NRS) — created from the Haskell scraper’s output.

Occasionally, we want to scrape a publication which is just a flat set of documents. E.g., not laws, but Official Opinions. In that case, I’m writing a traditional one-JSON-object-per-page scraper:

• A “flat” Scrapy scraper, in development, for the Opinions: https://github.com/public-law/oregon-administrative-rules-parser/blob/ga-ag-opinions/public_law/spiders/us/georgia_ag_opinions.py

So! You can see we’ve been trying several strategies — Haskell scrapers (running in Docker via a cron job) to Scrapy (running on Scraping Hub). But this looks like a great use case for Elixir / Beam / OTP.

@dogweather great! It would be great to know more about your use case, to see if we can help.

Otherwise also please be warned that Scrapy is far ahead of us in terms of stability and features. So it may be that a migration to Crawly will not be smooth, depending on your use case, and maybe will require patches to Crawly itself.

However from my side I can promiss my support with that things!

I’m really excited about giving Crawly a try. I’m using Scrapy currently, but I’d like to move away from Python.

Hey people,

I want to announce that I have released a new version of

Crawly

. The 0.8.0 contains a few important features like:

• retries support
• browser rendering (for JavaScript-based websites).

Hopefully, you could find it useful for your needs!

Btw here I have made another article about using Crawly with Tensorflow, which allows to organize machine learning project: Build Machine Learning Elixir Project

I wonder if it’s built with elixir? E.g. I did not find any information regarding how the framework works.

For web scraping, you can try this tool https://app.scrape.works/create-project to explore more scraping information.