Forum Moderators: open
crawler14.googlebot.com
crawler13.googlebot.com
crawler10.googlebot.com
crawler11.googlebot.com
crawler15.googlebot.com
crawler12.googlebot.com
Plus I notice two entries in my crawler identification logs:
Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Is Google pushing out loads of crawlers in an attempt to catch out cloaking/duplication etc. or are they just releasing more spiders to index the web faster?
Any ideas? Either way, between them, Yahoo and MSNbot they're starting to eat up a fair bit of band-width. I just had to increase the number of allowed connections as MSNbot crashed my server yesterday.
This has been discussed already.
The following thread is one that I am aware of:
[webmasterworld.com...]
I'm sure if you do a search you will find more.