Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?
Matt Probert
10:54 am on Mar 9, 2005 (gmt 0)
Robots are under no obligation to read or take notice of the requests in a robots.txt file. It is good manners for search engine robots to respect the site owner's wishes, as indicated in a robots.txt file, but that's all.
Matt
cooldoug
10:48 pm on Mar 9, 2005 (gmt 0)
Thanks for the reply, its clearer to me now.:)
Rajith
10:24 am on Mar 12, 2005 (gmt 0)
I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine.
vincevincevince
8:54 pm on Mar 26, 2005 (gmt 0)
In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.
