homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Bots using robots
Do they really have to?

 1:29 am on Mar 9, 2005 (gmt 0)

Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?


Matt Probert

 10:54 am on Mar 9, 2005 (gmt 0)

Robots are under no obligation to read or take notice of the requests in a robots.txt file. It is good manners for search engine robots to respect the site owner's wishes, as indicated in a robots.txt file, but that's all.



 10:48 pm on Mar 9, 2005 (gmt 0)

Thanks for the reply, its clearer to me now.:)


 10:24 am on Mar 12, 2005 (gmt 0)

I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine.


 8:54 pm on Mar 26, 2005 (gmt 0)

In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved