Welcome to WebmasterWorld Guest from 220.127.116.11 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor
Bots using robots Do they really have to? cooldoug msg:1526194 1:29 am on Mar 9, 2005 (gmt 0) Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?
Matt Probert msg:1526195 10:54 am on Mar 9, 2005 (gmt 0)
Robots are under no obligation to read or take notice of the requests in a robots.txt file. It is good manners for search engine robots to respect the site owner's wishes, as indicated in a robots.txt file, but that's all.
cooldoug msg:1526196 10:48 pm on Mar 9, 2005 (gmt 0)
Thanks for the reply, its clearer to me now.:) Rajith msg:1526197 10:24 am on Mar 12, 2005 (gmt 0)
I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine. vincevincevince msg:1526198 8:54 pm on Mar 26, 2005 (gmt 0)
In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.