homepage Welcome to WebmasterWorld Guest from 54.242.140.11
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Bots using robots
Do they really have to?
cooldoug

5+ Year Member



 
Msg#: 567 posted 1:29 am on Mar 9, 2005 (gmt 0)

Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?

 

Matt Probert

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 567 posted 10:54 am on Mar 9, 2005 (gmt 0)

Robots are under no obligation to read or take notice of the requests in a robots.txt file. It is good manners for search engine robots to respect the site owner's wishes, as indicated in a robots.txt file, but that's all.

Matt

cooldoug

5+ Year Member



 
Msg#: 567 posted 10:48 pm on Mar 9, 2005 (gmt 0)

Thanks for the reply, its clearer to me now.:)

Rajith

5+ Year Member



 
Msg#: 567 posted 10:24 am on Mar 12, 2005 (gmt 0)

I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine.

vincevincevince

WebmasterWorld Senior Member vincevincevince us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 567 posted 8:54 pm on Mar 26, 2005 (gmt 0)

In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved