homepage Welcome to WebmasterWorld Guest from 54.226.80.196
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    

Hope




msg:400948
 2:47 pm on Aug 2, 2000 (gmt 0)

I have been hit hard by Slurp in the last 3 months. At 3,000-6,000 pages a week, every week. The funny thing is I haven't submitted, and there are only about 2,000 pages to the site.

Slurp has been hitting the site about 3-5 times a week and doing a deep crawl each time. I am concerned about this. With all the crawling slurp is doing, could we be punished for some spam offence?

Also, why is slurp so happy with our site?

 

Air




msg:400949
 2:59 pm on Aug 2, 2000 (gmt 0)

Slurp will send multiple spiders for the same page, 3000-6000 hits by Slurp could mean that only 300 unique page requests were actually made.

Have a look at your logs to see if in fact this is the case.

littleman




msg:400950
 3:51 pm on Aug 2, 2000 (gmt 0)

Also, I have found the slurp out of Japan, 202.212.5.*, to come back very often and do heavy crawling. I think this only happens if you submit to Goo - there is evidence that they maintain an isolated db.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved