Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- June 2013 - Blocking Bad Spiders / Bots / User Agents
wilderness - 8:00 pm on Jun 22, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4586407.htm
See my reply in this recent thread [webmasterworld.com], which is a good beginning for black-listing, and is currently being used on a website.
See the thread Request Protocol [webmasterworld.com], which will deny many pests with a single line.
Brought to you by WebmasterWorld: http://www.webmasterworld.com