Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Bots using robots

Do they really have to?



1:29 am on Mar 9, 2005 (gmt 0)

10+ Year Member

Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?

Matt Probert

10:54 am on Mar 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Robots are under no obligation to read or take notice of the requests in a robots.txt file. It is good manners for search engine robots to respect the site owner's wishes, as indicated in a robots.txt file, but that's all.



10:48 pm on Mar 9, 2005 (gmt 0)

10+ Year Member

Thanks for the reply, its clearer to me now.:)


10:24 am on Mar 12, 2005 (gmt 0)

10+ Year Member

I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine.


8:54 pm on Mar 26, 2005 (gmt 0)

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member

In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.

Featured Threads

Hot Threads This Week

Hot Threads This Month