Welcome to WebmasterWorld Guest from 54.161.110.186

Forum Moderators: goodroi

Message Too Old, No Replies

Bots using robots

Do they really have to?

     
1:29 am on Mar 9, 2005 (gmt 0)

10+ Year Member



Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?
10:54 am on Mar 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Robots are under no obligation to read or take notice of the requests in a robots.txt file. It is good manners for search engine robots to respect the site owner's wishes, as indicated in a robots.txt file, but that's all.

Matt

10:48 pm on Mar 9, 2005 (gmt 0)

10+ Year Member



Thanks for the reply, its clearer to me now.:)
10:24 am on Mar 12, 2005 (gmt 0)

10+ Year Member



I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine.
8:54 pm on Mar 26, 2005 (gmt 0)

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member



In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month