homepage Welcome to WebmasterWorld Guest from 54.204.58.87
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
crawlX.googlebot.com vs. crawlerX.googlebot.com
what's the difference (if there is one)?
AfiComp




msg:125449
 2:58 am on Mar 1, 2003 (gmt 0)

Hi all,
Well - is there a difference between the two user agents?
There's also a difference it their IP address range.

10X in advance,
AfiComp

 

Batman




msg:125450
 3:32 am on Mar 1, 2003 (gmt 0)

As far as I can tell (plz correct me if I'm wrong), this is the deal:

64.68.82.* -> crawler##.googlebot.com a.k.a. Fast Crawl a.k.a. Freshbot
64.68.84.* -> crawl##.googlebot.com a.k.a. Deep Crawl a.k.a. Googlebot
216.239.46.* -> crawl##.googlebot.com a.k.a. Deep Crawl a.k.a. Googlebot

zarm777




msg:125451
 4:49 am on Mar 1, 2003 (gmt 0)

...so what are these then, freshbots?

crawl25.googlebot.com (64.68.80.157)
crawl25.googlebot.com (64.68.80.144)
crawl25.googlebot.com (64.68.80.150)
crawl25.googlebot.com (64.68.80.158)
crawl25.googlebot.com (64.68.80.149)

crawl23.googlebot.com (64.68.80.57)
crawl23.googlebot.com (64.68.80.56)
crawl23.googlebot.com (64.68.80.54)
crawl23.googlebot.com (64.68.80.53)

crawl24.googlebot.com (64.68.80.61)
crawl24.googlebot.com (64.68.80.69)
crawl24.googlebot.com (64.68.80.71)
crawl24.googlebot.com (64.68.80.75)
crawl24.googlebot.com (64.68.80.72)

crawl27.googlebot.com (64.68.80.201)
crawl27.googlebot.com (64.68.80.203)

crawl26.googlebot.com (64.68.80.159)

You know you are bored when... please update :o)

Marval




msg:125452
 6:38 am on Mar 1, 2003 (gmt 0)

I believe that actually any of the 64.68.x.x are Freshbots
the 216.239.x.x is the Deep Crawlers although recently there have been reports of bots being used interchangebly I havent seen it - hope that helps

kendos




msg:125453
 11:29 am on Mar 1, 2003 (gmt 0)

Here is the complete list :

Crawlx
---------
1216.239.46.1->216.239.46.30
2216.239.46.31 ->216.239.46.60
3216.239.46.61 ->216.239.46.80
4216.239.46.81 ->216.239.46.110
5216.239.46.111 ->216.239.46.140
6216.239.46.141 ->216.239.46.160
7216.239.46.161 ->216.239.46.189
8216.239.46.190 ->216.239.46.219
9216.239.46.220 ->216.239.46.240
1064.68.84.1 ->64.68.84.30
1164.68.84.31 ->64.68.84.60
1264.68.84.61 ->64.68.84.80
1364.68.84.129 ->64.68.84.158
1464.68.84.159 ->64.68.84.188
1564.68.84.189 ->64.68.84.208
1664.68.85.1 ->64.68.85.30
1764.68.85.31 ->64.68.85.60
1864.68.85.61 ->64.68.85.80
1964.68.85.129 ->64.68.85.158
2064.68.85.159 ->64.68.85.188
2164.68.85.189 ->64.68.85.208
2264.68.80.1 ->64.68.80.30
2364.68.80.31 ->64.68.80.60
2464.68.80.61 ->64.68.80.80
2564.68.80.129 ->64.68.80.158
2664.68.80.159 ->64.68.80.188
2764.68.80.189 ->64.68.80.208
2864.68.81.1 ->64.68.81.30
2964.68.81.31 ->64.68.81.60
3064.68.81.61 ->64.68.81.80

Crawler
-----------
164.68.86.1 ->64.68.86.30
234.68.86.31 ->64.68.86.60
364.68.86.61 ->64.68.86.80
464.68.86.130 ->64.68.86.158
564.68.86.159 ->64.68.86.188
664.68.86.189 ->64.68.86.208
764.68.87.1 ->64.68.87.30
864.68.87.31 ->64.68.87.60
964.68.87.61 ->64.68.87.80
1064.68.82.1 ->64.68.82.30
1164.68.82.31 ->64.68.82.60
1264.68.82.61 ->64.68.82.80
1364.68.82.81 ->64.68.82.110
1464.68.82.111 ->64.68.82.140
1564.68.82.141 ->64.68.82.160
1664.68.82.161 ->64.68.82.189
1764.68.82.190 ->64.68.82.219
1864.68.82.220 ->64.68.82.240

AfiComp




msg:125454
 3:06 pm on Mar 1, 2003 (gmt 0)

10X all :)

AfiComp

zarm777




msg:125455
 3:43 pm on Mar 1, 2003 (gmt 0)

Just so I am clear on this, you are saying that...

crawler11.googlebot.com would be a freshbot
and
crawl23.googlebot.com would be a deepbot?

I was always looking at the ip's before but this does look like it makes much more sense as I look at my logs.

Thanks...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved