homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How to ban inktomisearch.com's bots?
I have been hit hard with so many requests by this.

 5:21 pm on Apr 11, 2004 (gmt 0)

I think this(inktomisearch.com slurp) is from pure inktomi, right? Isn't the Yahoo Slurp use different names?

I use AWStats and inktomisearch.com slurp has been hitting my sites so hard that it knocked down the site a few times already in the past few weeks.

The site is ranked well on Pure Inktomi and Yahoo as well as Google.

I don't want to lose the traffic from Yahoo and Google but it is OK to lose the traffic from MSN or others that use pure Inktomi feed, since it is less than 10% of total SE traffic.

I don't like it crashing my site with more than 10,000 hits a week to my site.

So, my question is "If I ban
inktomisearch.com's slurp, am I banning Yahoo slurp, too or just pure inktomi slup?"



 5:22 pm on Apr 11, 2004 (gmt 0)

And what user agent name do I have to use to ban the pure inktomi slup in the robots.txt?


 5:24 pm on Apr 11, 2004 (gmt 0)

The site is a forum site with tens of thousand pages with one file name "forum.asp" with all the parameters after question mark for the category ID and post ID, etc. So, If I ban the slurp for the "forum.asp" it will ban the slurp from all my forum pages.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved