Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- WebDataCentreBot


incrediBILL - 8:15 pm on Jul 6, 2008 (gmt 0)



Since I whitelist Google, Yahoo, MSN, they go through first and then anything with "http://" in the UA is blocked so stuff like that is easily stopped.

Of course it has also blocked a couple of browser plug-ins in the past which in one case caused a small riot and got the "http://" removed from the UA of that plug-in ;)


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3686831.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com