Forum Moderators: open

Message Too Old, No Replies

Multiple Googlebots

Is this a duplicate content/cloaking filter?

         

internetheaven

12:08 pm on Jul 28, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



My logs used to show just one googlebot, now there are loads, here are the ones to hit me today:

crawler14.googlebot.com
crawler13.googlebot.com
crawler10.googlebot.com
crawler11.googlebot.com
crawler15.googlebot.com
crawler12.googlebot.com

Plus I notice two entries in my crawler identification logs:

Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)

Is Google pushing out loads of crawlers in an attempt to catch out cloaking/duplication etc. or are they just releasing more spiders to index the web faster?

Any ideas? Either way, between them, Yahoo and MSNbot they're starting to eat up a fair bit of band-width. I just had to increase the number of allowed connections as MSNbot crashed my server yesterday.

internetheaven

10:04 pm on Jul 28, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Does no-one have any comments on this or am I posting in the wrong thread?

I figured the Google forum would give better info on Googlebot as opposed to the Crawler forum.

phpmaven

11:40 pm on Jul 28, 2004 (gmt 0)

10+ Year Member



internetheaven

This has been discussed already.

The following thread is one that I am aware of:
[webmasterworld.com...]

I'm sure if you do a search you will find more.