Forum Moderators: open

Message Too Old, No Replies

Crawlers

which is which?

         

Webber

4:43 am on Mar 3, 2004 (gmt 0)

10+ Year Member



Does anyone know which crawlers are fulfilling which purposes?

In my stats I see the following hostnames:

crawler2.googlebot.com
crawler4.googlebot.com
crawler8.googlebot.com
crawler9.googlebot.com

Are these all doing the same things or do these crawlers have different 'jobs'?
Adsense, Fresh- or Deepcrawl?

Powdork

7:54 am on Mar 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Not sure anymore. There used to be crawlx.googlebot and crawlerx.googlebot and one was deep and one was fresh. Not so anymore, they're multitasking except that Mediapartners is Adsense.