Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

msnbot Spoofing Browser ID

10:12 pm on Feb 11, 2011 (gmt 0)

New User

5+ Year Member

joined:Feb 11, 2011
votes: 0

I recently became suspicious that msnbot was ignoring robots.txt and spoofing a human browser ID on some of my sites. The following code was installed in Apache at a test:

# Deny msnbot IP block access to any file
# when it's not being honest in its browser ID field
RewriteCond %{REMOTE_ADDR} ^207\.46\.12\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\.195\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\.199\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\.204\.
RewriteCond %{HTTP_USER_AGENT} !msnbot [NC]
RewriteRule .* /msnbot.html [L]

This delivers a unique page to anything emanating from an msnbot block that is not identifying itself as msnbot.

After searching Bing today for the terms in the test page, sure enough, there it was. Why msnbot should feel that it's exempt from robots.txt and properly identifying itself, I can't imagine.
6:17 am on Feb 16, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member themadscientist is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 14, 2008
votes: 62

Hi Brock,
Welcome to WebmasterWorld!

Those guys at M$ are a bit nutty, and I have seen IP addresses resolve to M$ corporate surfing as a regular browser, but I'm not sure if I've ever seen a bot do it personally.

I have heard people have access to their IPs to use at times, but I'm not really sure I've experienced them not following the rules personally. Maybe there's some other reason they're visting the page or being redirected and it just seems like they're totally spoofing their UA?

Uh, you know they're spidering as bingbot now, which would correctly be sent to that page when coming from one of those IP ranges, right? ;) I made this text small, because most people don't read this stuff, especially if I put enough of it to make it look like it's work for them to figure out what I'm trying to say. Hope you enjoy your stay here at WebmasterWorld. Personally, I really enjoy the knowledge and discussions.
4:00 am on Feb 17, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member

joined:Feb 16, 2007
votes: 0

The UA should be "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

And you're missing some ranges: - (no rDNS) - msnbot-65-52-ccc-ddd.search.msn.com