Gollum - Robots.txt parser and cache

Your Libraries & OS Mentoring Libraries
2

Heh, that could be quite useful for the scrapers around here. ^.^

3 Likes