homepage Welcome to WebmasterWorld Guest from 54.166.108.167
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robozilla and Robots.txt
Brett_Tabke




msg:1528014
 12:23 pm on May 3, 2001 (gmt 0)

Does robozilla obey robots.txt? And if so, what does it do if it runs into a url in the database, that is also blocked by a robots.txt at the site?

 

Mike_Mackin




msg:1528015
 12:31 pm on May 3, 2001 (gmt 0)

Robozilla visits all the links within the Open Directory periodically, marking the ones that return errors for review. It does not spider the site.

[dmoz.org...]

[robotstxt.org...]

Brett_Tabke




msg:1528016
 9:27 am on May 4, 2001 (gmt 0)

Yep, that is what it does. Doesn't answer the question. :-)

Does it read robots.txt? (I've always thought not, but someone in another group is claiming it does).

[robotstxt.org...]

Brett_Tabke




msg:1528017
 8:11 am on May 5, 2001 (gmt 0)

Robo answered the question himself (actually it was a her). No, robo doesn't read robots.txt.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved