homepage Welcome to WebmasterWorld Guest from 54.196.62.23
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
bingbot joins the 131.253 party
lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4490524 posted 9:30 pm on Sep 1, 2012 (gmt 0)

There are two recent threads about msnbot-media visiting from the 131.253.41. range:

[webmasterworld.com...] (1 month ago)

[webmasterworld.com...] (2 months ago)

I've had them set to Ignore all along, because they're well-behaved on my site. (ymmv, as shown in those earlier threads). Yesterday for the first time I met the ordinary bingbot from the same neighborhood:

131.253.47.251 - - [31/Aug/2012:21:33:15 -0700] "GET /robots.txt HTTP/1.1" 200 911 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
131.253.47.251 - - [31/Aug/2012:22:36:07 -0700] "GET /fun/AlonzoMelissa.html HTTP/1.1" 200 10423 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"


That's .47. while all the mediabots have used .41. in the same range. But again, unmistakable bingbot behavior, with more requests for robots.txt than for pages.

Huh.

D'you suppose it's got something to do with the spike in bingbot activity reported elsewhere? 157.55. got too crowded so they had to outsource?

:: minor detour here to cross-check search and discover that amazon has the brazen gall to label something as "copyrighted material" although it is scraped from Project Gutenberg and therefore by definition in the public domain ::

 

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4490524 posted 9:38 pm on Sep 1, 2012 (gmt 0)

It returns the proper rDNS msnbot-131-253-47-251.search.msn.com

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4490524 posted 8:06 pm on Sep 2, 2012 (gmt 0)

Lucy, you missed my posting here... [webmasterworld.com...]

It gives all the ranges (approx) that I could get from DNS.

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4490524 posted 3:13 am on Sep 3, 2012 (gmt 0)

Today I got a swathe of bingbots from that range

Yup, I did miss that. Best excuse: you were heroically trying to stay OT while the rest of the thread got
:: cough-cough ::
a bit sidetracked.

Philosophically interesting, though. Robot A has never offended me, while Robot B is firmly in Shoot to Kill territory. The next guy comes along and reports the exact opposite experience.

And, uhm, yes, I did mistype the topic header. S/b 131.253 of course, as in the body of all relevant posts. That would be the range Microsoft is subletting from Avaya, not to be confused with the range Amazon is subletting from Merck.

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 4490524 posted 5:40 am on Sep 3, 2012 (gmt 0)

Has anyone done a reverse DNS scan thru all those IP ranges to catalog which IPs claim to be specifically for bingbot?

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4490524 posted 8:48 pm on Sep 3, 2012 (gmt 0)

The 131.253 range? Yes. I ran a DNS scan 29th Aug. The ranges I posted elsewhere are fairly accurate - possibly about 20% IPs not used for bot (that's "intelligent guesswork" without actually checking).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved