Welcome to WebmasterWorld Guest from 188.8.131.52 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe to WebmasterWorld
Hope msg:400948 2:47 pm on Aug 2, 2000 (gmt 0) I have been hit hard by Slurp in the last 3 months. At 3,000-6,000 pages a week, every week. The funny thing is I haven't submitted, and there are only about 2,000 pages to the site.
Slurp has been hitting the site about 3-5 times a week and doing a deep crawl each time. I am concerned about this. With all the crawling slurp is doing, could we be punished for some spam offence?
Also, why is slurp so happy with our site?
Air msg:400949 2:59 pm on Aug 2, 2000 (gmt 0)
Slurp will send multiple spiders for the same page, 3000-6000 hits by Slurp could mean that only 300 unique page requests were actually made.
Have a look at your logs to see if in fact this is the case.
littleman msg:400950 3:51 pm on Aug 2, 2000 (gmt 0)
Also, I have found the slurp out of Japan, 202.212.5.*, to come back very often and do heavy crawling. I think this only happens if you submit to Goo - there is evidence that they maintain an isolated db.