homepage Welcome to WebmasterWorld Guest from 54.226.43.155
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
MSNbot changing to Bingbot on Oct.1, 2010
incrediBILL




msg:4161286
 1:02 pm on Jun 29, 2010 (gmt 0)

Per the Bing blog [bing.com]:
On October 1st, 2010, we will drop the beta designation from the Bing crawler and change the name of the crawler to reflect Microsoft's new brand for search. Instead of the old msnbot 2.0b showing up in your server logs, the updated user agent will be:

Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)

The HTTP header From field will also change as shown below:

From: msnbot(at)microsoft.com

will become

From: bingbot(at)microsoft.com

So anyone herding msnbot using .htaccess or other means will need to start adjusting your software accordingly.

However, don't worry about your robots.txt file however, because they plan to retain backwards compatibility.
We want webmasters to know that bingbot will still honor robots.txt directives written for msnbot, so no change is required to your robots.txt file(s).


There was no official mention about whether their rDNS used for bot validation would change so I suspect it will still retain the following format:
msnbot-aaa-bbb-ccc-ddd.search.msn.com

 

ByronM




msg:4161494
 5:42 pm on Jun 29, 2010 (gmt 0)

about time ;)

dstiles




msg:4161676
 10:24 pm on Jun 29, 2010 (gmt 0)

Shame the important part of the UA no longer starts at the beginning of the string. Takes just a bit longer to parse. :(

What about the media bots etc? Are they going to be exactly the same UA? If so how will we know to block the things?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved