Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Automatically detect crawlers / bots
incrediBILL - 11:47 pm on Dec 8, 2011 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4395123.htm
I tend to block about 300 bots/day, most of them are the same 200 that keep knocking day after day. This is down from the heyday of my site when there were nearly 1K things blocked per day, it was insane!
Brought to you by WebmasterWorld: http://www.webmasterworld.com