Forum Moderators: open

Message Too Old, No Replies

Been hit hard this month!

All major SE bots visited this month

         

creative craig

4:19 pm on Apr 30, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Just had a good look at the raw logs for this month and:

Inktomi deepcrawled the site twice
Googlebot deepcrawled the site, over a period of three days
Looksmart has been stopping by every day for a few pages at a time
Scooter has taken the whole site as well
FAST took half the site in the last week

I know alot of sites get hit hard every day (webmasterworld for example), but has anyone else seen this much activity, or am I the lucky one ;)?

Craig

leoo24

6:26 pm on Apr 30, 2003 (gmt 0)

10+ Year Member



hey craig :)

i've been getting hit pretty hard too,
here's how many visits i've had from:

1. inktomisearch.com 254
2. googlebot.com 156
3. fastsearch.net 83
4. av.com 77

DavidT

5:40 pm on May 5, 2003 (gmt 0)

10+ Year Member



I can add AskJeeves/Teoma to that list, has been coming in about once a week and methodically going through big chunks of the site each time.

creative craig

12:47 pm on May 6, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The Microsoft prototype crawler has also been coming back daily to take a few pages at each visit, better keep an eye on this one :)

Craig