How do you know if it's a freshbot or deepcrawl?

64.68... or 216.239...

[irock]

Hi, I would like to know how to identify a Google freshbot and a deepcrawl spider. I read from previous post that 64.68 is a freshbot and 216.239 is a deepcrawl. Can anyone confirm?

Thanks!

[nancyb]

try this thread [webmasterworld.com...]

all kinds of googlebot info

[Sasquatch]

Nope, no one can confirm.

I would not be at all surprised if during the deep crawl if they were all involved in the deep crawl, and during the rest of the month, some of the deep crawl bots were turned over to playing freshbot if they weren't needed for anything else.

They use general purpose machines that network boot with no local storage (or at least that is the way it was a few years ago) so the machines have no reason to be for a specific purpose.

[NameNick]

Sasquatch,

Your deep-crawl-bot-turnover theory sounds plausible for me and I had the same thoughts since I saw the crawlXX and the crawlerXX bots occuring mixed in my log files this morning.

NN

[Sasquatch]

If you check your logs, you might even notice that not all the crawlXX bots are active with freshbot activity and a couple of crawlerXX are.

All they need to do is change the image that they load on the machine and it quickly changes from an incredibly efficient deepbot to a freshbot to a indexer to a searcher. They can even do custom tweaks to the OS to make it the most efficient for that task. All it takes is a quick change to your DHCP and boot server, and even those would be easy to automate with load scheduling.