WebGuerrilla - 2:59 am on Feb 28, 2001 (gmt 0)

"In our head content we have among other content the followign for robots:
<meta name="Robots" content="index">
<meta name="Robots" content="follow">"

These meta tags probably don't cause any harm, but they have absolutely no effect when it comes to how often your site gets spidered, or how many pages get crawled. Some engines do honor the noindex meta tag, but others will ignore it. All credible engines honor the robots.txt. That is why you will see so many requests for it in your logs. Before a spider begins to crawl, it will check that file to see if there are any pages it shouldn't index.