Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- amazonaws.com plays host to wide variety of bad bots
GaryK - 7:26 pm on Mar 9, 2009 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/3828718.htm
I do the same, Sam. At least I do on most of my sites. Cowbot slipped through on one of my domains last week and it's now operating from EC2. I just started a new thread about it.
Brought to you by WebmasterWorld: http://www.webmasterworld.com