Jump to letter: [
3ABCDEFGHIJKLMNOPQRSTUVWXYZ
]
perl-WWW-RobotRules-Parser - Just Parse robots.txt
- Description:
WWW::RobotRules::Parser allows you to simply parse robots.txt files as
described in http://www.robotstxt.org/wc/norobots.html. Unlike
WWW::RobotRules (which is very cool), this module does not take into
consideration your user agent name when parsing. It just parses the
structure and returns a hash containing the whole set of rules. You can
then use this to do whatever you like with it.
Packages