Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- And Now Google's Doing It. JS Stats Show GoogleBot


Samizdata - 10:45 pm on May 14, 2011 (gmt 0)


Far as I can tell, 64.233.x.x, 72.14.x.x, 74.125.x.x are strictly Google Web Preview

Nothing is ever so straightforward in the Google botworld.

72.14.x.x includes Google Wireless Transcoder, Google Translate and Feedfetcher-Google.

74.125.x.x includes Google-Sitemaps and Google Keyword Tool. It is also used for automated checks on disallowed files (so that GoogleBot can claim to honour the robots protocol).

Some here may block them all, but others allow some or all of them.

Google Web Preview was introduced in typical Google land grab fashion with a threat to punish those who served their own visual preview.

All your base are belong to Mountain View.

...


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4312058.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com