homepage Welcome to WebmasterWorld Guest from 54.205.254.108
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to ban inktomisearch.com's bots?
I have been hit hard with so many requests by this.
itisgene




msg:1528787
 5:21 pm on Apr 11, 2004 (gmt 0)

I think this(inktomisearch.com slurp) is from pure inktomi, right? Isn't the Yahoo Slurp use different names?

I use AWStats and inktomisearch.com slurp has been hitting my sites so hard that it knocked down the site a few times already in the past few weeks.

The site is ranked well on Pure Inktomi and Yahoo as well as Google.

I don't want to lose the traffic from Yahoo and Google but it is OK to lose the traffic from MSN or others that use pure Inktomi feed, since it is less than 10% of total SE traffic.

I don't like it crashing my site with more than 10,000 hits a week to my site.

So, my question is "If I ban
inktomisearch.com's slurp, am I banning Yahoo slurp, too or just pure inktomi slup?"

 

itisgene




msg:1528788
 5:22 pm on Apr 11, 2004 (gmt 0)

And what user agent name do I have to use to ban the pure inktomi slup in the robots.txt?

itisgene




msg:1528789
 5:24 pm on Apr 11, 2004 (gmt 0)

The site is a forum site with tens of thousand pages with one file name "forum.asp" with all the parameters after question mark for the category ID and post ID, etc. So, If I ban the slurp for the "forum.asp" it will ban the slurp from all my forum pages.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved