homepage Welcome to WebmasterWorld Guest from 54.237.78.165
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
msnbot Spoofing Browser ID
Brock Samson



 
Msg#: 4265899 posted 10:12 pm on Feb 11, 2011 (gmt 0)

I recently became suspicious that msnbot was ignoring robots.txt and spoofing a human browser ID on some of my sites. The following code was installed in Apache at a test:

# Deny msnbot IP block access to any file
# when it's not being honest in its browser ID field
RewriteCond %{REMOTE_ADDR} ^207\.46\.12\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\.195\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\.199\. [OR]
RewriteCond %{REMOTE_ADDR} ^207\.46\.204\.
RewriteCond %{HTTP_USER_AGENT} !msnbot [NC]
RewriteRule .* /msnbot.html [L]

This delivers a unique page to anything emanating from an msnbot block that is not identifying itself as msnbot.

After searching Bing today for the terms in the test page, sure enough, there it was. Why msnbot should feel that it's exempt from robots.txt and properly identifying itself, I can't imagine.

 

TheMadScientist

WebmasterWorld Senior Member themadscientist us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4265899 posted 6:17 am on Feb 16, 2011 (gmt 0)

Hi Brock,
Welcome to WebmasterWorld!

Those guys at M$ are a bit nutty, and I have seen IP addresses resolve to M$ corporate surfing as a regular browser, but I'm not sure if I've ever seen a bot do it personally.

I have heard people have access to their IPs to use at times, but I'm not really sure I've experienced them not following the rules personally. Maybe there's some other reason they're visting the page or being redirected and it just seems like they're totally spoofing their UA?

Uh, you know they're spidering as bingbot now, which would correctly be sent to that page when coming from one of those IP ranges, right? ;) I made this text small, because most people don't read this stuff, especially if I put enough of it to make it look like it's work for them to figure out what I'm trying to say. Hope you enjoy your stay here at WebmasterWorld. Personally, I really enjoy the knowledge and discussions.

caribguy

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4265899 posted 4:00 am on Feb 17, 2011 (gmt 0)

The UA should be "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

And you're missing some ranges:
157.54.0.0 - 157.60.255.255 (no rDNS)
65.52.0.0 - 65.55.255.255 msnbot-65-52-ccc-ddd.search.msn.com

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved