- Crawler, Spider, and User Agent ID forum at WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

Hope

2:47 pm on Aug 2, 2000 (gmt 0)

I have been hit hard by Slurp in the last 3 months. At 3,000-6,000 pages a week, every week. The funny thing is I haven't submitted, and there are only about 2,000 pages to the site.

Slurp has been hitting the site about 3-5 times a week and doing a deep crawl each time. I am concerned about this. With all the crawling slurp is doing, could we be punished for some spam offence?

Also, why is slurp so happy with our site?

Air

2:59 pm on Aug 2, 2000 (gmt 0)

Slurp will send multiple spiders for the same page, 3000-6000 hits by Slurp could mean that only 300 unique page requests were actually made.

Have a look at your logs to see if in fact this is the case.

littleman

3:51 pm on Aug 2, 2000 (gmt 0)

Also, I have found the slurp out of Japan, 202.212.5.*, to come back very often and do heavy crawling. I think this only happens if you submit to Goo - there is evidence that they maintain an isolated db.

Hope

Air

littleman

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week