Mozilla/4.0

MSN strikes (out) again.

[Pfui]

What fresh hell is this?

65.55.234.160
Mozilla/4.0

robots.txt? YES

[GaryK]

I don't appear to get a rDNS for that IP Address. But it seems to belong to MS Corporate and not msnbot per se. Is that what you're seeing too?

What kinds of files did it take? Was it well behaved? What about referrers?

[dstiles]

I'm getting a lot of junk from MSN at present, much of it with no UA at all. Whether they hit robots or not I haven't checked: if it looks like a hyena it's treated like a hyena.

I have that IP listed as being in a range that returned rDNS not so long ago. Today I removed several others from my accredited list that I'm sure resolved before. Could it be MSN is rearranging its IPs into more sensible blocks and purging non-bot rDNS and giving it over to proxy use? No, don't laugh. It's always possible. :)

For reference: the IP block the hits came in on today were 207.68.133.nnn.

[Pfui]

@GaryK, if I hadn't seen the "Mozilla/4.0" hit from 65.55.234.160 while using a little Apache access_log tail script, I'd have missed it. Only asked for/took robots.txt; no refs.

We do rDNS/double-lookup at the webserver level so an IP in an access_log = no rDNS. Skimming some older notes, neighboring 65.55.234.169 ran "msnbot-media/1.1" in May, 2008.

[dstiles]

Forgot to mention: almost all ^Mozilla/4.0$ UAs I see are from oriental IPs. MSN is one of very few I've seen outside of those ranges.

I wonder if this is microsoft at all. Perhaps they are from China or similar using MS proxies. If so there is no indication they really are proxies, at least not by the usual headers.

[Pfui]

Same host on 07-21 and 07-22, confirming Microsoft's running "Mozilla/4.0":

msnbot-65-55-230-228.search.msn.com
Mozilla/4.0

robots.txt? YES

[keyplyr]

65.55.108.170 - - [31/Jul/2009:15:20:01 -0700] "GET www.example/images/file.jpg HTTP/1.1" 200 2666 "-" "Mozilla/4.0"

robots.txt: yes

Took 9 image files

[wilderness]

keyplr,
Were the image files disallowed in robots.txt?
TIA

Don

[keyplyr]

Hi Don,

No the images file taken were not disallowed. No bad behavior, other than the sneakiness of the UA string.

[wilderness]

keyplr,
I tacked this on the other MSN thread, however even this was NOT technically a violation of robots.txt [webmasterworld.com]

[Pfui]

cosmos.cosmosblu.search.live.net
Mozilla/4.0

robots.txt? YES

Prior UA from this dances-with-msn.com's-other-bots bot:

cosmos.cosmosblu.search.live.net
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)

robots.txt? YES