Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor
How to ban inktomisearch.com's bots? I have been hit hard with so many requests by this. itisgene msg:1528787 5:21 pm on Apr 11, 2004 (gmt 0) I think this(inktomisearch.com slurp) is from pure inktomi, right? Isn't the Yahoo Slurp use different names?
I use AWStats and inktomisearch.com slurp has been hitting my sites so hard that it knocked down the site a few times already in the past few weeks.
The site is ranked well on Pure Inktomi and Yahoo as well as Google.
I don't want to lose the traffic from Yahoo and Google but it is OK to lose the traffic from MSN or others that use pure Inktomi feed, since it is less than 10% of total SE traffic.
I don't like it crashing my site with more than 10,000 hits a week to my site.
my question is "If I ban inktomisearch.com's slurp, am I banning Yahoo slurp, too or just pure inktomi slup?"
itisgene msg:1528788 5:22 pm on Apr 11, 2004 (gmt 0)
And what user agent name do I have to use to ban the pure inktomi slup in the robots.txt? itisgene msg:1528789 5:24 pm on Apr 11, 2004 (gmt 0)
The site is a forum site with tens of thousand pages with one file name "forum.asp" with all the parameters after question mark for the category ID and post ID, etc. So, If I ban the slurp for the "forum.asp" it will ban the slurp from all my forum pages.