homepage Welcome to WebmasterWorld Guest from 50.19.206.49
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
bingbot joins the 131.253 party
lucy24




msg:4490526
 9:30 pm on Sep 1, 2012 (gmt 0)

There are two recent threads about msnbot-media visiting from the 131.253.41. range:

[webmasterworld.com...] (1 month ago)

[webmasterworld.com...] (2 months ago)

I've had them set to Ignore all along, because they're well-behaved on my site. (ymmv, as shown in those earlier threads). Yesterday for the first time I met the ordinary bingbot from the same neighborhood:

131.253.47.251 - - [31/Aug/2012:21:33:15 -0700] "GET /robots.txt HTTP/1.1" 200 911 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
131.253.47.251 - - [31/Aug/2012:22:36:07 -0700] "GET /fun/AlonzoMelissa.html HTTP/1.1" 200 10423 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"


That's .47. while all the mediabots have used .41. in the same range. But again, unmistakable bingbot behavior, with more requests for robots.txt than for pages.

Huh.

D'you suppose it's got something to do with the spike in bingbot activity reported elsewhere? 157.55. got too crowded so they had to outsource?

:: minor detour here to cross-check search and discover that amazon has the brazen gall to label something as "copyrighted material" although it is scraped from Project Gutenberg and therefore by definition in the public domain ::

 

incrediBILL




msg:4490528
 9:38 pm on Sep 1, 2012 (gmt 0)

It returns the proper rDNS msnbot-131-253-47-251.search.msn.com

dstiles




msg:4490717
 8:06 pm on Sep 2, 2012 (gmt 0)

Lucy, you missed my posting here... [webmasterworld.com...]

It gives all the ranges (approx) that I could get from DNS.

lucy24




msg:4490794
 3:13 am on Sep 3, 2012 (gmt 0)

Today I got a swathe of bingbots from that range

Yup, I did miss that. Best excuse: you were heroically trying to stay OT while the rest of the thread got
:: cough-cough ::
a bit sidetracked.

Philosophically interesting, though. Robot A has never offended me, while Robot B is firmly in Shoot to Kill territory. The next guy comes along and reports the exact opposite experience.

And, uhm, yes, I did mistype the topic header. S/b 131.253 of course, as in the body of all relevant posts. That would be the range Microsoft is subletting from Avaya, not to be confused with the range Amazon is subletting from Merck.

incrediBILL




msg:4490812
 5:40 am on Sep 3, 2012 (gmt 0)

Has anyone done a reverse DNS scan thru all those IP ranges to catalog which IPs claim to be specifically for bingbot?

dstiles




msg:4491033
 8:48 pm on Sep 3, 2012 (gmt 0)

The 131.253 range? Yes. I ran a DNS scan 29th Aug. The ranges I posted elsewhere are fairly accurate - possibly about 20% IPs not used for bot (that's "intelligent guesswork" without actually checking).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved