Inktomisearch bot attack

I have over 15 inktomi bots everyday on my website

[meetzah]

Hello,

I don't know if there is the spot to post this but I don't know what is happening: now I have 16 inktomi bots crawling my website. Google bot and msn are om my website as well but as normal. Here are the datas from my logs regarding the inktomi bots I was telling you about:

72.30.129.113 lj601863.inktomisearch.com
72.30.252.98 lj602278.inktomisearch.com
72.30.226.156 lj602362.inktomisearch.com
72.30.252.162 lj602238.inktomisearch.com
68.142.249.170 lj2160.inktomisearch.com
72.30.131.205 lj601915.inktomisearch.com
72.30.252.160 lj602236.inktomisearch.com
72.30.103.27 lj601570.inktomisearch.com
etc....

I must tell you that I don't get any traffic from inktomy and from yahoo just few hits per day. My domain is not a .com one and my website is not comeercial. I never advertised with inktomy or yahoo.

Should I bann the bot using .htaccess or robots.txt?

Thank you.

[Tim]

I would rate limit the crawl to your site to slow it down. If this doesn't work please stickymail me your domain and I can have someone look at it.
Thanks, Tim

There is a Yahoo! Slurp-specific extension to robots.txt which allows you to set a lower limit on our crawler request rate.

You can add a "Crawl-delay: xx" instruction, where "xx" is a delay value between successive crawler accesses. If the crawler rate is a problem for your server, you can set the delay up to 5 or 10 or a comfortable value for your server.

Setting a crawl-delay of 10 for Yahoo! Slurp would look something like:

User-agent: Slurp
Crawl-delay: 10

[edited by: Tim at 4:27 pm (utc) on July 16, 2006]