- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- GoogleBot/2.1
incrediBILL - 11:30 pm on Jun 7, 2006 (gmt 0)
|Yeah but the point is that we think people are spoofing a user agent and routing through a google-owned proxy |
If Google permits that, then they get what they deserve, not a lot we can do about it unless they cough up a definitive list of crawler IPs.
However, even if they were, most people don't use NOARCHIVE and the server caches your pages anyway and the content can be had there as well.
Thread source: http://www.webmasterworld.com/search_engine_spiders/3273.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com