Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Automatically detect crawlers / bots


incrediBILL - 11:47 pm on Dec 8, 2011 (gmt 0)


I tend to block about 300 bots/day, most of them are the same 200 that keep knocking day after day. This is down from the heyday of my site when there were nearly 1K things blocked per day, it was insane!


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4395123.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com