@dogweather great! It would be great to know more about your use case, to see if we can help.
Otherwise also please be warned that Scrapy is far ahead of us in terms of stability and features. So it may be that a migration to Crawly will not be smooth, depending on your use case, and maybe will require patches to Crawly itself.
However from my side I can promiss my support with that things!
Excellent - we have several scrapers running nightly, and are continually writing more, to extract online laws from various countries and locations. My startup, https://public.law, makes the laws easier to read and search.
I use scrapers to create JSON representations of each legal resource. Usually, a collection of laws are hierarchical - a tree of e.g., Divisions > Chapters > Sections. And so I like to write scrapers which return a single JSON tree because it’s easier to capture the parent/child relationships while the spider is running, than to try to recreate it on import.
The final result, after import, is a website like this: Nevada Revised Statutes (NRS) — created from the Haskell scraper’s output.
Occasionally, we want to scrape a publication which is just a flat set of documents. E.g., not laws, but Official Opinions. In that case, I’m writing a traditional one-JSON-object-per-page scraper:
So! You can see we’ve been trying several strategies — Haskell scrapers (running in Docker via a cron job) to Scrapy (running on Scraping Hub). But this looks like a great use case for Elixir / Beam / OTP.
Hey, @dogweather I just wonder how it goes? E.g. I wonder if you have tried it out? Did you have any issues?
From our side, I have asked my colleagues to have a glance at the target you have shared. Taking into account that crawly can’t save data in a nested JSON, I have advised them to use a flat structure, injecting all fields into one basic item.
So far our results are far away from being ideal 
see: http://crawlyui.com/spider?spider=Elixir.Spiders.OregonRegs
I wonder if you tried it from your side, and if you have managed to gather more items than we did.
Hey people, sorry for restoring this a bit old topic again, but I want to add another library called CrawlyUI to the discussion.
You might wonder why I decided to add another library here, as well as there are plenty here already. Well, I believe it’s somehow different as:
You can find the deployed version here: http://crawlyui.com/
And a short explanation on how to create a spider here: https://www.youtube.com/watch?v=1O-XArob5rU&ab_channel=OlegTarasenko
Really hope you will like it!
