Welcome to WebmasterWorld Guest from 54.147.44.93

Forum Moderators: mack

Message Too Old, No Replies

msnbot Spoofing Browser ID

     

Brock Samson

10:12 pm on Feb 11, 2011 (gmt 0)



I recently became suspicious that msnbot was ignoring robots.txt and spoofing a human browser ID on some of my sites. The following code was installed in Apache at a test:

# Deny msnbot IP block access to any file
# when it's not being honest in its browser ID field
RewriteCond %{REMOTE_ADDR} ^207\.46\.12\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\.195\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\.199\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\.204\.
RewriteCond %{HTTP_USER_AGENT} !msnbot [NC]
RewriteRule .* /msnbot.html [L]

This delivers a unique page to anything emanating from an msnbot block that is not identifying itself as msnbot.

After searching Bing today for the terms in the test page, sure enough, there it was. Why msnbot should feel that it's exempt from robots.txt and properly identifying itself, I can't imagine.

TheMadScientist

6:17 am on Feb 16, 2011 (gmt 0)

WebmasterWorld Senior Member themadscientist is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Hi Brock,
Welcome to WebmasterWorld!

Those guys at M$ are a bit nutty, and I have seen IP addresses resolve to M$ corporate surfing as a regular browser, but I'm not sure if I've ever seen a bot do it personally.

I have heard people have access to their IPs to use at times, but I'm not really sure I've experienced them not following the rules personally. Maybe there's some other reason they're visting the page or being redirected and it just seems like they're totally spoofing their UA?

Uh, you know they're spidering as bingbot now, which would correctly be sent to that page when coming from one of those IP ranges, right? ;) I made this text small, because most people don't read this stuff, especially if I put enough of it to make it look like it's work for them to figure out what I'm trying to say. Hope you enjoy your stay here at WebmasterWorld. Personally, I really enjoy the knowledge and discussions.

caribguy

4:00 am on Feb 17, 2011 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



The UA should be "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

And you're missing some ranges:
157.54.0.0 - 157.60.255.255 (no rDNS)
65.52.0.0 - 65.55.255.255 msnbot-65-52-ccc-ddd.search.msn.com
 

Featured Threads

Hot Threads This Week

Hot Threads This Month