homepage Welcome to WebmasterWorld Guest from 54.167.249.155
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 31 message thread spans 2 pages: < < 31 ( 1 [2]     
Now seeing Bingbot
Testing the waters a few days early
jdMorgan




msg:4208509
 3:12 am on Sep 29, 2010 (gmt 0)

Spotted in the logs today.

207.46.195.227 Tue Sep 28 20:24:15 2010 "GET /widgets.html HTTP/1.1" 200 4321 "-" "Mozilla/5.0 (compatible; bingbot/2.0 +http://www.bing.com/bingbot.htm)" Connection="Keep-Alive" Accept="*/*" Accept-Encoding="gzip, deflate"

Jim

 

Pfui




msg:4229278
 5:59 pm on Nov 11, 2010 (gmt 0)

Still hitting bare (no rDNS), but at least asked for robots.txt this time:

157.55.16.231
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

robots.txt? Yes

Hmm. Bare MSN IPs get the standard disallow version of robots.txt, the official .search.msn.com/msnbot/bingbot bots don't. Yet another IP/UA variation/combination to code for? MSN, Google 's new Web Preview UA [webmasterworld.com...] -- it's never-ending, I tellya.

This 31 message thread spans 2 pages: < < 31 ( 1 [2]
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved