Silvery - 3:02 pm on Jun 11, 2007 (gmt 0)

meetzah2, incrediBILL's right - if you ban search engine bots by IP addresses, you can ruin your rankings in the search engine. Likewise, banning them completely via robots.txt could do the same.

Even though you don't fully explain the problem you're experiencing, I'm going to go out on a limb and assume that perhaps your server is having problems keeping up with too-frequent requests from Slurp?

If you're really just trying to reduce the frequency of the spider requests, Yahoo! allows you to specify a "Crawl-delay" in your robots.txt file.

You could set the delay from 5 to 10 to space out the requests coming into you. The search engines need to be able to request your pages in order to index your content and make if findable through their results for your users, but they don't want to request pages so frequently as to become a defacto denial-of-service attack. So, unless you have some reason you don't want your website/pages to be found on the internet, just limit the crawl rate and don't ban anything.

See Yahoo!'s help section for more details:

[help.yahoo.com...]