Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- June 2013 - Blocking Bad Spiders / Bots / User Agents
Key_Master - 10:44 pm on Jun 22, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4586407.htm
Whitelisting is the way to go, but you can't just apply it to one header. All the headers have to be parsed for accuracy. It also requires a fair amount of knowledge about browsers and the headers each one uses.
Brought to you by WebmasterWorld: http://www.webmasterworld.com