| Robozilla and Robots.txt
|
Brett_Tabke
msg:1528014 | 12:23 pm on May 3, 2001 (gmt 0) |
Does robozilla obey robots.txt? And if so, what does it do if it runs into a url in the database, that is also blocked by a robots.txt at the site?
|
Mike_Mackin
msg:1528015 | 12:31 pm on May 3, 2001 (gmt 0) |
Robozilla visits all the links within the Open Directory periodically, marking the ones that return errors for review. It does not spider the site. [dmoz.org...] [robotstxt.org...]
|
Brett_Tabke
msg:1528016 | 9:27 am on May 4, 2001 (gmt 0) |
Yep, that is what it does. Doesn't answer the question. :-) Does it read robots.txt? (I've always thought not, but someone in another group is claiming it does). [robotstxt.org...]
|
Brett_Tabke
msg:1528017 | 8:11 am on May 5, 2001 (gmt 0) |
Robo answered the question himself (actually it was a her). No, robo doesn't read robots.txt.
|
|
|
