Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Reasons for using Googlebot user agent?
jmccormac - 9:25 am on Mar 20, 2010 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4100609.htm
Scrapers use this in an attempt to bypass controls. Some sites will whitelist the large search engine bots but apply access controls and limits to ordinary browsers.
Brought to you by WebmasterWorld: http://www.webmasterworld.com