Span - 5:39 pm on Jun 23, 2005 (gmt 0)

You can control the rate at which your site is crawled by using the Crawl-Delay feature. The Crawl-Delay feature allows you to specify the number of seconds between visits to your site. Note that it may take quite a long time to crawl a site if there are many pages and the Crawl-Delay is set high. You could specify an interval of 30 seconds between requests with an entry like this: User-agent: BecomeBot Crawl-Delay: 30 Disallow: /cgi-bin