dimitarvp

dimitarvp

I don’t mean to sound unethical but I’d only be interested in such a library / platform if I could opt out of respecting robots.txt (meaning: not respecting it). There are a number of business mediator and governmental websites that haven’t touched their robots.txt files in 10+ years but many businesses need the publicly available data and scrape them daily regardless.

So, is there a way to ignore robots.txt?

2
Post #11
mischov

mischov

It looks like robots.txt is handled by a middleware, so it’s likely that you can run without that middleware if you want (though I haven’t confirmed that this is true).

1
Post #12
oltarasenko

oltarasenko OP

Yes, sure. You can ignore it. Obviously no one should recommend it in a tutorial, right :slight_smile: ?

2
Post #13
oltarasenko

oltarasenko OP

Yes, true! Thanks for the hint!

0
Post #14
oltarasenko

oltarasenko OP

Yes true. I also did not have problems installing Rust. However it still feels like it’s better to use something which does not require anything for the sake of getting started guide?

0
Post #15
dimitarvp

dimitarvp

LOL :003: Good point. Thank you!

1
Post #16
dimitarvp

dimitarvp

50/50. I actually agree with you but in a world where people just assume you already have Node.JS installed in order to do anything, I would guess also assuming they have Rust is not a stretch. :slight_smile: (especially having in mind how brainlessly easy it is to install)

Was a minor remark anyway, ignore it. It was mostly aimed at the tutorial being Windows-friendly, with which Rust helps a lot. Nothing else.

2
Post #17
oltarasenko

oltarasenko OP

Yeah, but on the other hand it always makes sense to lower the curve :slight_smile:

2
Post #18
oltarasenko

oltarasenko OP

Also wanted to add, that I am looking for new production usages of Crawly or CrawlyUI. If you want to try the toolset I am suggesting, I will be happy to help. Just write here, or via issues on github!

3
Post #19
dogweather

dogweather

What JSON output does Crawly support? With some sites, I like Scrapy’s default style of a JSON object per page. But with other sites, I want to create a single JSON tree representing the site. This is possible with Scrapy with a few tricks. How hard would this be to do with Crawly?

0
Post #20