Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Amazon AWS Hosts Bad Bots

Pfui - 2:12 pm on Oct 19, 2011 (gmt 0)


Two seconds apart to the same rarely directly-hit file. Coincidence?

ec2-204-236-161-233.us-west-1.compute.amazonaws.com
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)

02:38:30 /dir/filename.html
robots.txt? NO

ec2-50-16-74-139.compute-1.amazonaws.com
Mozilla/5.0 (compatible; Topicmarks/1.0)

02:38:32 /dir/filename.html
robots.txt? NO

Diffbot (old-timer): [google.com...]
Topicmarks (just posted): [webmasterworld.com...]

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4368965.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com