Is this a real MSN bot or a fake?

IP seems OK, but is overly zealous.

[grandma genie]

I have been seeing this msn bot in my server logs for days and it is coming in like a tidal wave. Is this really MSN? Could it be something else? It is part of the Microsoft block of IPs. I've just never seen it so active before.

65.55.25.nnn - - [07/Sep/2010:04:10:49 -0400] "GET (image file) HTTP/1.1" 200 60451 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._"

Grandma_genie

[Pfui]

There are threads galore in this forum about MSN's bots (ditto IP addresses), one thread of which is a current "Featured Home Page Discussion" (my first; yay!) and includes details about the exact same UA.

FWIW, I find it helpful and time-saving to scan thread titles (and/or search) before posting about a new or odd UA I've spotted because chances are someone's already posted about it. Sometimes even m'self. (slaps head)

[grandma genie]

Hi Pfui - Am now reading your very interesting Featured Home Page Discussion. I know it seems I do not check the site first before posting, but I actually do search both Google, Project HoneyPot and Webmasterworld before I post. The part of the user_agent string that was bothering me was the strange little ._ at the end of each line. It stood out so much in the server logs and there were so many of them. It just was not acting like the usual MSN bot. So I started to worry that someone was hiding behind the IP, but didn't know how to find out, so that is when I posted. I have to assume a person who manages their own server has more resources for checking their logs. That is why I come here for help. I can't go to my host. They try to be helpful, but I know they are limited in their knowledge. Thank you for all your help. I hope I'm not being a pest.

[grandma genie]

Hello again,
The msnbot is really starting to bug me. I get msnbot from two different IPs. The one that used to show up was 207.46.195.nnn (the last three digits change). But the one that is just going crazy (half my logs are this one) is 65.55.25.nnn and the IP stays the same. I am considering just banning the one IP address. I want to be indexed by msn, but don't want that crazy bot with the 65.55.25.nnn hogging the logs. How does the forum feel about banning the one IP? - Grandma_genie

[dstiles]

All activity I've seen from that IP range has no valid rDNS so the range 65.55.25/24 is blocked. The bot that shows up on that range here has the underscore suffix in the UA.

[Pfui]

If yours is a newish and/or smallish (<50 pages) site, and/or you're reading raw access logs, what you describe as "going crazy" may very well be normal. MSN's bots can be very active (if not terribly accurate, results-wise). On my largest site, msnbot's hit 1,715 times this month. By comparison, Google's hit 800 times.

Bottom Line: You block msnbot from .search.msn.com's IPs at your peril, imho. But if you want to, the thread I mentioned -- [webmasterworld.com...] -- includes comments about how others currently handle MSN's assorted IPs. There are prior threads on the topic, too.

[grandma genie]

Hi Mr. D and Pfui -- The IP range is now blocked. I have a huge site and it is 10 years old. I have more than 1000 hits from that IP today alone. So whoever that was, they are now blocked. The other msn IP that visits (207.46.195.nnn) behaves itself. Thanks guys.

[dstiles]

In passing, I've picked up another msnbot range today - probably been active for several days though not months. I picked it up when the underscore bot hit.

65.55.55.199 - 65.55.55.221

[grandma genie]

I'm also getting visits from 65.52.49.nn. But this one, so far, is behaving.

[jmccormac]

There is a possibility that this is a legit spider. There has been an upswing of Bing traffic to a few of my sites recently beginning in August. Bing is ramping up in the battle against Google.

Regards...jmcc

[grandma genie]

Oh, oh... Now what is this?:

65.55.3.nnn - - [28/Sep/2010:20:00:02 -0400] "GET /stuff.html HTTP/1.1" 200 11722 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._"
65.55.3.nnn - - [28/Sep/2010:20:00:22 -0400] "GET /morestuff.html HTTP/1.1" 200 10392 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._"

This is the same dot underscore but the IP resolves to (msnbot-65-55-3-nnn.search.msn.com.) doing a reverse dns. If it goes crazy like its cousin I'm going to have to block it, too. What do you think, Mr. D?

[dstiles]

Whitelist the "real" msnbot IPs BUT block the undesirable bots such as underscore.

Some folk hereabouts do IP rDNS lookups on the fly but not all of us can do that so have to create and maintain IP ranges for bots.