Welcome to WebmasterWorld Guest from 54.146.239.96

Forum Moderators: goodroi

Message Too Old, No Replies

Bots using robots

Do they really have to?

     

cooldoug

1:29 am on Mar 9, 2005 (gmt 0)

10+ Year Member



Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?

Matt Probert

10:54 am on Mar 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Robots are under no obligation to read or take notice of the requests in a robots.txt file. It is good manners for search engine robots to respect the site owner's wishes, as indicated in a robots.txt file, but that's all.

Matt

cooldoug

10:48 pm on Mar 9, 2005 (gmt 0)

10+ Year Member



Thanks for the reply, its clearer to me now.:)

Rajith

10:24 am on Mar 12, 2005 (gmt 0)

10+ Year Member



I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine.

vincevincevince

8:54 pm on Mar 26, 2005 (gmt 0)

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member



In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.