Welcome to WebmasterWorld Guest from 54.198.222.129

Forum Moderators: open

Message Too Old, No Replies

crawlX.googlebot.com vs. crawlerX.googlebot.com

what's the difference (if there is one)?

     
2:58 am on Mar 1, 2003 (gmt 0)

10+ Year Member



Hi all,
Well - is there a difference between the two user agents?
There's also a difference it their IP address range.

10X in advance,
AfiComp

3:32 am on Mar 1, 2003 (gmt 0)

10+ Year Member



As far as I can tell (plz correct me if I'm wrong), this is the deal:

64.68.82.* -> crawler##.googlebot.com a.k.a. Fast Crawl a.k.a. Freshbot
64.68.84.* -> crawl##.googlebot.com a.k.a. Deep Crawl a.k.a. Googlebot
216.239.46.* -> crawl##.googlebot.com a.k.a. Deep Crawl a.k.a. Googlebot

4:49 am on Mar 1, 2003 (gmt 0)

10+ Year Member



...so what are these then, freshbots?

crawl25.googlebot.com (64.68.80.157)
crawl25.googlebot.com (64.68.80.144)
crawl25.googlebot.com (64.68.80.150)
crawl25.googlebot.com (64.68.80.158)
crawl25.googlebot.com (64.68.80.149)

crawl23.googlebot.com (64.68.80.57)
crawl23.googlebot.com (64.68.80.56)
crawl23.googlebot.com (64.68.80.54)
crawl23.googlebot.com (64.68.80.53)

crawl24.googlebot.com (64.68.80.61)
crawl24.googlebot.com (64.68.80.69)
crawl24.googlebot.com (64.68.80.71)
crawl24.googlebot.com (64.68.80.75)
crawl24.googlebot.com (64.68.80.72)

crawl27.googlebot.com (64.68.80.201)
crawl27.googlebot.com (64.68.80.203)

crawl26.googlebot.com (64.68.80.159)

You know you are bored when... please update :o)

6:38 am on Mar 1, 2003 (gmt 0)

10+ Year Member



I believe that actually any of the 64.68.x.x are Freshbots
the 216.239.x.x is the Deep Crawlers although recently there have been reports of bots being used interchangebly I havent seen it - hope that helps
11:29 am on Mar 1, 2003 (gmt 0)

10+ Year Member



Here is the complete list :

Crawlx
---------
1216.239.46.1->216.239.46.30
2216.239.46.31 ->216.239.46.60
3216.239.46.61 ->216.239.46.80
4216.239.46.81 ->216.239.46.110
5216.239.46.111 ->216.239.46.140
6216.239.46.141 ->216.239.46.160
7216.239.46.161 ->216.239.46.189
8216.239.46.190 ->216.239.46.219
9216.239.46.220 ->216.239.46.240
1064.68.84.1 ->64.68.84.30
1164.68.84.31 ->64.68.84.60
1264.68.84.61 ->64.68.84.80
1364.68.84.129 ->64.68.84.158
1464.68.84.159 ->64.68.84.188
1564.68.84.189 ->64.68.84.208
1664.68.85.1 ->64.68.85.30
1764.68.85.31 ->64.68.85.60
1864.68.85.61 ->64.68.85.80
1964.68.85.129 ->64.68.85.158
2064.68.85.159 ->64.68.85.188
2164.68.85.189 ->64.68.85.208
2264.68.80.1 ->64.68.80.30
2364.68.80.31 ->64.68.80.60
2464.68.80.61 ->64.68.80.80
2564.68.80.129 ->64.68.80.158
2664.68.80.159 ->64.68.80.188
2764.68.80.189 ->64.68.80.208
2864.68.81.1 ->64.68.81.30
2964.68.81.31 ->64.68.81.60
3064.68.81.61 ->64.68.81.80

Crawler
-----------
164.68.86.1 ->64.68.86.30
234.68.86.31 ->64.68.86.60
364.68.86.61 ->64.68.86.80
464.68.86.130 ->64.68.86.158
564.68.86.159 ->64.68.86.188
664.68.86.189 ->64.68.86.208
764.68.87.1 ->64.68.87.30
864.68.87.31 ->64.68.87.60
964.68.87.61 ->64.68.87.80
1064.68.82.1 ->64.68.82.30
1164.68.82.31 ->64.68.82.60
1264.68.82.61 ->64.68.82.80
1364.68.82.81 ->64.68.82.110
1464.68.82.111 ->64.68.82.140
1564.68.82.141 ->64.68.82.160
1664.68.82.161 ->64.68.82.189
1764.68.82.190 ->64.68.82.219
1864.68.82.220 ->64.68.82.240

3:06 pm on Mar 1, 2003 (gmt 0)

10+ Year Member



10X all :)

AfiComp

3:43 pm on Mar 1, 2003 (gmt 0)

10+ Year Member



Just so I am clear on this, you are saying that...

crawler11.googlebot.com would be a freshbot
and
crawl23.googlebot.com would be a deepbot?

I was always looking at the ip's before but this does look like it makes much more sense as I look at my logs.

Thanks...

 

Featured Threads

Hot Threads This Week

Hot Threads This Month