homepage Welcome to WebmasterWorld Guest from 54.145.209.80
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    

Hope

10+ Year Member



 
Msg#: 97 posted 2:47 pm on Aug 2, 2000 (gmt 0)

I have been hit hard by Slurp in the last 3 months. At 3,000-6,000 pages a week, every week. The funny thing is I haven't submitted, and there are only about 2,000 pages to the site.

Slurp has been hitting the site about 3-5 times a week and doing a deep crawl each time. I am concerned about this. With all the crawling slurp is doing, could we be punished for some spam offence?

Also, why is slurp so happy with our site?

 

Air

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 97 posted 2:59 pm on Aug 2, 2000 (gmt 0)

Slurp will send multiple spiders for the same page, 3000-6000 hits by Slurp could mean that only 300 unique page requests were actually made.

Have a look at your logs to see if in fact this is the case.

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 97 posted 3:51 pm on Aug 2, 2000 (gmt 0)

Also, I have found the slurp out of Japan, 202.212.5.*, to come back very often and do heavy crawling. I think this only happens if you submit to Goo - there is evidence that they maintain an isolated db.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved