Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

2:47 pm on Aug 2, 2000 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 2, 2000
votes: 0

I have been hit hard by Slurp in the last 3 months. At 3,000-6,000 pages a week, every week. The funny thing is I haven't submitted, and there are only about 2,000 pages to the site.

Slurp has been hitting the site about 3-5 times a week and doing a deep crawl each time. I am concerned about this. With all the crawling slurp is doing, could we be punished for some spam offence?

Also, why is slurp so happy with our site?


2:59 pm on Aug 2, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 10, 2000
votes: 0

Slurp will send multiple spiders for the same page, 3000-6000 hits by Slurp could mean that only 300 unique page requests were actually made.

Have a look at your logs to see if in fact this is the case.

3:51 pm on Aug 2, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
votes: 0

Also, I have found the slurp out of Japan, 202.212.5.*, to come back very often and do heavy crawling. I think this only happens if you submit to Goo - there is evidence that they maintain an isolated db.