I know many of you blindly accept Google's IP ranges as trustworthy and this is hardly the case. If you're allowing anything with a Google IP range global access then all you're doing is allowing a certain subset of scrapers carte blanche access because of their IP.
The Google IP range hosts all sorts of tools that be used for nefarious purposes including:
- Google Wireless Transcoder
- Google Translator
- Google Engine
Luckily the Google Engine forces all requests to have an "AppEngine-Google” prefix that can be easily filtered.
Plus, I've seen the old proxy hijacking, which I thought that much like polio and had been eliminated, rear it's ugly head once again. The only wayt to stop this is to verify Googlebot is only crawling from it's valid IP addresses.
Full trip Googlebot validation is a must have front line defense, use it!
[edited by: incrediBILL at 6:31 pm (utc) on Dec 18, 2013]