Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Google IPs Cannot Be Trusted
incrediBILL - 10:58 am on Dec 18, 2013 (gmt 0)Google Wireless Transcoder
I know many of you blindly accept Google's IP ranges as trustworthy and this is hardly the case. If you're allowing anything with a Google IP range global access then all you're doing is allowing a certain subset of scrapers carte blanche access because of their IP.
The Google IP range hosts all sorts of tools that be used for nefarious purposes including:
Luckily the Google Engine forces all requests to have an "AppEngine-Google” prefix that can be easily filtered.
Plus, I've seen the old proxy hijacking, which I thought that much like polio and had been eliminated, rear it's ugly head once again. The only wayt to stop this is to verify Googlebot is only crawling from it's valid IP addresses.
Full trip Googlebot validation is a must have front line defense, use it!
[edited by: incrediBILL at 6:31 pm (utc) on Dec 18, 2013]
Thread source:: http://www.webmasterworld.com/search_engine_spiders/4631778.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com