Deep Crawl Madness

What's he doing in there?!

[trillianjedi]

We've had five googlebots deepcrawling our site since last night (where they spent a happy couple of hours) and this morning (have been there for the last 3 hours).

This site is not big by any stretch (maybe 100 pages max).

What's he up to? Following links out, indexing another site and then back to us? Presumably if that's the case, a good sign?

TJ

[BGumble]

One PR7 site here has thousands of pages already in the index and more waiting. Deepbot is also being very, very, very slow and could never get it all at the pace it's at now. Last update I'd have dozens of the buggers, now it's one every few minutes.

[trillianjedi]

Well, we (still) have 87 trapped googlebots in our site maxing out the 3mbits pipe.

I don't know how many bots google has (I suspect thousands) but having this many all stuck in my site can't be helping anyone.

It's gone through another couple of hundred meg since my last post. I hope google pull the stops out and sort this soon.

I'm not worried about the money (it's only a few pounds) - it's been worth it to see googlebot deepcrawl about 2,000,000 of my pages, 1,999,650 or so of which don't actually exist!

Poor things.

TJ

[rfgdxm1]

Perhaps they decided Googlebot needed to go on a diet? Except for the one that trillianjedi has around, which is a glutton. ;)

[trillianjedi]

Do you want to see something really funny? Is it ok if I post the guest list for my forum here? It's ok, it's all googlebot (all 85 or so guests!). I'll delete the keywords and anything domain related anyway because I don't want anyone else hitting the site.

If you do know my domain, please don't hit it, it's suffering enough as it is and I do have legit members on there too (I will also delete)..... give me one sec.

TJ

[trillianjedi]

OK, this is for a five minute period about 7 minutes ago (server is in Europe and an hour behind me in the UK). I've "search and deleted" all domain and other info, but the IP's are all there and all googlebots. This is about half of it and it's been like this for a day or two now:-

Guest Fri Apr 18, 2003 11:15 pm Fri Apr 18, 2003 11:15 pm 216.239.46.206
Guest Fri Apr 18, 2003 11:15 pm Fri Apr 18, 2003 11:15 pm 216.239.46.136
Guest Fri Apr 18, 2003 11:15 pm Fri Apr 18, 2003 11:15 pm 216.239.46.136
Guest Fri Apr 18, 2003 11:15 pm Fri Apr 18, 2003 11:15 pm 216.239.46.105
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.88
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.77
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.30
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.96
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.77
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.222
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.193
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.220
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.118
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.3
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.220
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.60
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.164
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.220
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.128
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.86
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.75
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.20
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.55
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.140
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.236
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.193
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.41
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.82
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.164
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.167
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.53
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.102
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.128
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.60
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.105
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.27
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.98
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.128
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.20
Guest Fri Apr 18, 2003 11:14 pm Fri Apr 18, 2003 11:14 pm 216.239.46.172
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.42
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.147
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.206
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.184
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.33
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.184
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.63
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.82
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.12
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.77
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.63
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.223
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.171
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.148
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.53
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.75
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.173
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.41
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.96
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.236
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.77
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.48
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.136
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.20
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.76
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.23
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.134
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.27
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.193
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.12
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.206
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.19
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.36
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.172
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.27
Guest Fri Apr 18, 2003 11:13 pm Fri Apr 18, 2003 11:13 pm 216.239.46.36
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.33
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.133
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.53
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.223
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.23
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.60
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.67
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.48
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.98
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.121
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.98
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.167
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.105
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.184
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.33
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.165
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.48
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.76
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.133
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.118
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.66
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.223
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.101
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.42
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.20
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.102
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.236
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.140
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.172
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.165
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.66
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.136
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.99
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.66
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.98
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.27
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.12
Guest Fri Apr 18, 2003 11:12 pm Fri Apr 18, 2003 11:12 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.41
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.147
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.86
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.27
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.104
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.167
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.167
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.43
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.23
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.133
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.41
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.82
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.172
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.147
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.148
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.171
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.193
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.220
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.41
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.124
Guest Fri Apr 18, 2003 10:53 pm Fri Apr 18, 2003 11:11 pm 62.49.16.63
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.82
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.165
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.82
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.184
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.165
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.206
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.147
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.30
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.82
Guest Fri Apr 18, 2003 11:11 pm Fri Apr 18, 2003 11:11 pm 216.239.46.76
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.55
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.3
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.100
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.12
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.171
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.30
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.67
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.85
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.85
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.153
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.76
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.42
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.98
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.172
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.3
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.171
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.12
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.12
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.43
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.121
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.147
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.220
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.133
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.20
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.133
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.41
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.19
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.172
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.102
Guest Fri Apr 18, 2003 11:10 pm Fri Apr 18, 2003 11:10 pm 216.239.46.172

[BGumble]

04:01 PM crawl4.googlebot.com
04:02 PM crawl1.googlebot.com
04:16 PM crawl5.googlebot.com
04:19 PM crawl4.googlebot.com
04:21 PM crawl5.googlebot.com

You have thousands and thousands more to go mr.bot and will never finish eating if you just nibble. Btw.. I'm not panicking, just curious at the pace!

[uci_bink]

Yea I dont know whats going on with the pace of googlebot...at this rate it will take him until the next update to get all of my site...I wish he was maxing out my pipe like he is for Jedi.

Oh well....guess all we can do is watch and wait!

-Kevin