Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- June 2013 - Blocking Bad Spiders / Bots / User Agents
wilderness - 12:51 pm on Jun 24, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4586407.htm
|- Using the header information from above, white list based on user agent, Accept-Encoding, Connection, and Accept headers when there are no proxy headers. |
Here's one example for headers [webmasterworld.com], however I don't see any mention of Proxy.
Brought to you by WebmasterWorld: http://www.webmasterworld.com