| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
|Robozilla and Robots.txt|
| 12:23 pm on May 3, 2001 (gmt 0)|
Does robozilla obey robots.txt? And if so, what does it do if it runs into a url in the database, that is also blocked by a robots.txt at the site?
| 12:31 pm on May 3, 2001 (gmt 0)|
Robozilla visits all the links within the Open Directory periodically, marking the ones that return errors for review. It does not spider the site.
| 9:27 am on May 4, 2001 (gmt 0)|
Yep, that is what it does. Doesn't answer the question. :-)
Does it read robots.txt? (I've always thought not, but someone in another group is claiming it does).
| 8:11 am on May 5, 2001 (gmt 0)|
Robo answered the question himself (actually it was a her). No, robo doesn't read robots.txt.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved