Welcome to WebmasterWorld Guest from 54.196.243.192

Forum Moderators: goodroi

Message Too Old, No Replies

Bots using robots

Do they really have to?

     
1:29 am on Mar 9, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 15, 2005
posts:65
votes: 0


Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?
10:54 am on Mar 9, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 11, 2004
posts:1014
votes: 0


Robots are under no obligation to read or take notice of the requests in a robots.txt file. It is good manners for search engine robots to respect the site owner's wishes, as indicated in a robots.txt file, but that's all.

Matt

10:48 pm on Mar 9, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 15, 2005
posts:65
votes: 0


Thanks for the reply, its clearer to me now.:)
10:24 am on Mar 12, 2005 (gmt 0)

New User

10+ Year Member

joined:Feb 1, 2005
posts:16
votes: 0


I agree with Matt Probert, Roborts are under no obligation but its good to have robot.txt in the site which give clear idea to robot of the search engine.
8:54 pm on Mar 26, 2005 (gmt 0)

Senior Member from MY 

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 1, 2003
posts:4847
votes: 0


In many cases, the robots.txt file is an advantage for the robot. For example, keeping it out of never ending CGI loops.