- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Why robots.txt if you want everything indexed?
lorax - 2:06 pm on Mar 3, 2003 (gmt 0)
|... it is also widely used for keeping the bad bots out altogether. |
Only if the bot requests the robots.txt file and not all of them do.
The robots.txt file is good for those bots that do request it and in those cases you can tell them which directories to stay out of - like an image directory or your cgi directory.
Thread source: http://www.webmasterworld.com/robots_txt/33.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com