Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Bots using robots

Do they really have to?



1:29 am on Mar 9, 2005 (gmt 0)

10+ Year Member

Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?

Matt Probert

10:54 am on Mar 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Robots are under no obligation to read or take notice of the requests in a robots.txt file. It is good manners for search engine robots to respect the site owner's wishes, as indicated in a robots.txt file, but that's all.



10:48 pm on Mar 9, 2005 (gmt 0)

10+ Year Member

Thanks for the reply, its clearer to me now.:)


10:24 am on Mar 12, 2005 (gmt 0)

10+ Year Member

I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine.


8:54 pm on Mar 26, 2005 (gmt 0)

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member

In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.