Does it help to get your site included in search engines or is it supposed to get rid of unnecessary linkage to other parts of your site?
stargeek
6:35 pm on Sep 28, 2005 (gmt 0)
you can use it to block access to sections you don't want indexed, like pages that may be considered duplicate content (printable pages), and other parts you don't want the se's to crawl, like shopping carts and suchlike.
Dijkgraaf
10:40 pm on Sep 28, 2005 (gmt 0)
No it doesn't help your site to be included in search engines, when a search engine ask for a robots.txt file, they have allready discovered a URL of your site that they are about to ask for. :-)
Some reason to use robots.txt are limiting bandwidth use by bots stopping bots requesting cgi and other scripts stopping bots requesting pages that no longer exist
