Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- MSN's many cloaked bots. Again.

Pfui - 4:53 pm on Aug 27, 2011 (gmt 0)


The following two --

bare-IP (no rDNS)
cloaked-UA
same-second
no-robots.txt

-- hits by MSN were part of a post-tweet swarm to the exact same file:

65.52.6.105
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
08/27 09:34:25 /dir2/filename.html

65.52.17.111
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
08/27 09:34:25 /dir2/filename.html

Why any major SE needs to repeatedly abuse the very rules they expect us to follow is beyond me. I guess it's 'because they can.'

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4182830.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com