homepage Welcome to WebmasterWorld Guest from 184.73.52.98
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Microsoft bot 157 ranges updated
DNS for IP ranges 157.55 and 157.56 updated
dstiles




msg:4384613
 11:05 pm on Nov 7, 2011 (gmt 0)

I've been keeping a desultory eye on the MS 157 ranges for some time. Not sure when it actually happened - probably within the past few weeks - but MS seem to have taken notice of complaints - or just finally got their act together.

I had a handful of IPs tagged as msnbot in my database from a previous DNS scan some time back. The new ranges/sub-ranges (about 100) extend this range but also, in some cases, extend or reduce previously logged ranges. The list I now have, taken from DNS scans of the MS DNS servers, is included below. I think the list is accurate but I cannot guarantee it.

I may have missed a few breaks in the actual bot list IPs (eg there may be a gap in a sequence I missed) but it is essentially correct.

------------------------------------------
157.55.2.138 - 157.55.2.182
157.55.7.138 - 157.55.7.182
157.55.10.202 - 157.55.10.246
157.55.11.138 - 157.55.11.182
157.55.11.200 - 157.55.11.244
157.55.13.10 - 157.55.13.116
157.55.13.138 - 157.55.13.182
157.55.16.9 - 157.55.16.99
157.55.16.140 - 157.55.16.231
157.55.17.73 - 157.55.17.117
157.55.17.137 - 157.55.17.228
157.55.18.9 - 157.55.18.53
157.55.18.74 - 157.55.18.119
157.55.18.138 - 157.55.18.183
157.55.18.202 - 157.55.18.251
157.55.19.10 - 157.55.19.55
157.55.19.74 - 157.55.19.119
157.55.19.138 - 157.55.19.183
157.55.19.202 - 157.55.19.251
157.55.20.10 - 157.55.20.55
157.55.20.74 - 157.55.20.183
157.55.20.202 - 157.55.20.247
157.55.21.10 - 157.55.21.55
157.55.21.74 - 157.55.21.119
157.55.21.138 - 157.55.21.183
157.55.21.202 - 157.55.21.252
157.55.22.10 - 157.55.22.55
157.55.22.74 - 157.55.22.119
157.55.22.138 - 157.55.22.183
157.55.22.202 - 157.55.22.247
157.55.23.10 - 157.55.23.55
157.55.23.74 - 157.55.23.183
157.55.36.139 - 157.55.36.228
157.55.37.10 - 157.55.37.103
157.55.37.139 - 157.55.37.228
157.55.38.10 - 157.55.38.99
157.55.38.139 - 157.55.38.236
157.55.39.10 - 157.55.39.99
157.55.39.139 - 157.55.39.228
157.55.48.7 - 157.55.48.51
157.55.48.71 - 157.55.48.115
157.55.48.135 - 157.55.48.243
157.55.50.7 - 157.55.50.51
157.55.50.71 - 157.55.50.115
157.55.98.8 - 157.55.98.52
157.55.98.74 - 157.55.98.118
157.55.99.10 - 157.55.99.54
157.55.99.199 - 157.55.99.243
157.55.100.10 - 157.55.100.54
157.55.102.7 - 157.55.102.51
157.55.103.135 - 157.55.103.179
157.55.103.199 - 157.55.103.243
157.55.106.10 - 157.55.106.13
157.55.106.74 - 157.55.106.118
157.55.106.138 - 157.55.106.182
157.55.106.202 - 157.55.106.246
157.55.107.10 - 157.55.107.102
157.55.107.138 - 157.55.107.182
157.55.107.202 - 157.55.107.246
157.55.108.10 - 157.55.108.118
157.55.108.138 - 157.55.108.182
157.55.108.202 - 157.55.108.246
157.55.109.10 - 157.55.109.118
157.55.109.138 - 157.55.109.182
157.55.109.202 - 157.55.109.246
157.55.110.10 - 157.55.110.115
157.55.110.135 - 157.55.110.179
157.55.110.202 - 157.55.110.246
157.55.111.74 - 157.55.111.118
157.55.111.138 - 157.55.111.139
157.55.111.202 - 157.55.111.246
157.55.112.199 - 157.55.112.243
157.55.114.10 - 157.55.114.54
157.55.114.202 - 157.55.114.241
157.55.116.8 - 157.55.116.97
157.55.118.74 - 157.55.118.113
157.55.118.138 - 157.55.118.177
157.55.154.135 - 157.55.154.224
------------------------------------------
157.56.0.10 - 157.56.0.99
157.56.0.139 - 157.56.0.228
157.56.1.10 - 157.56.1.99
157.56.1.135 - 157.56.1.184
157.56.2.10 - 157.56.2.99
157.56.2.185 - 157.56.2.228
157.56.3.7 - 157.56.3.94
157.56.3.135 - 157.56.3.222
157.56.4.7 - 157.56.4.94
157.56.4.135 - 157.56.4.222
157.56.5.7 - 157.56.5.94
157.56.5.135 - 157.56.5.222
157.56.16.135 - 157.56.16.178
157.56.16.199 - 157.56.16.242
157.56.17.7 - 157.56.17.50
157.56.17.71 - 157.56.17.114
157.56.17.135 - 157.56.17.178
157.56.17.199 - 157.56.17.242
157.56.18.7 - 157.56.18.50
157.56.18.135 - 157.56.18.222
157.56.80.7 - 157.56.80.51
157.56.80.71 - 157.56.80.115
------------------------------------------

 

keyplyr




msg:4384633
 11:58 pm on Nov 7, 2011 (gmt 0)

MS seem to have taken notice of complaints

Not sure if this is related but I've been complaining to Bing, msnbot, et al about errors these bots create at my server; sometimes as many as a couple hundred per day. I've exchanged numerous emails with them over the last few months.

Yesterday all those errors stopped.

dstiles




msg:4384662
 1:47 am on Nov 8, 2011 (gmt 0)

Could be it was that recent. I did a single check on a repeatedly failed IP on Saturday and that showed the update.

I complained to bingdude some time back, in another forum on WebmasterWorld, about the IP and UA problems and nudged again when he popped up there a few days ago so maybe that did some good. Who knows? :(

Pfui




msg:4384865
 1:48 pm on Nov 8, 2011 (gmt 0)

I'm curious: What do you do with the 157. info? Do you only allow msnbot/bingbot from those ranges? Or keep track for your own interest? Or--?

wilderness




msg:4384954
 5:16 pm on Nov 8, 2011 (gmt 0)

here's dstiles 156 & 157 Class C's (the lines are too many IMO to separate by Class D's) to regex:

157\.55\.([27]|[1[0136-9]|2[0-3]|3[6-9]|48|50|9[89]|10[0236-9]|11[012468]|154)\.

157\.56\.([0-5]|1[678]|80)\.

dstiles




msg:4385090
 11:39 pm on Nov 8, 2011 (gmt 0)

Pfui - I hold the ranges in a MySQL database, wherein are all other blockable ranges, short-term auto-blocks etc. When I detect a bingbot UA I check it against the IP. If there is no match, no access. Which is why I'd run up a massive block list of 157 IPs until this list got added.

Reason for the method is historical as much as anything - my security system is a decade or more old, built when IIS had no proper regex let alone a decent htaccess capability.

Pfui




msg:4385097
 12:12 am on Nov 9, 2011 (gmt 0)

Thanks, dstiles. I, too, must juggle expediency with capability (the server's; my own:) Thus when it comes to the 157s et al, things get a bit-ham-fisted: msnbot and bingbot are only okay from:

RewriteCond %{REMOTE_HOST} !\.(bing|live|msn)\.com$
RewriteCond %{REMOTE_ADDR} !^65\.(54|55)\.
RewriteCond %{REMOTE_ADDR} !^157\.55\.
RewriteCond %{REMOTE_ADDR} !^207\.46\.

(The jury's still out on 157.56.)

motorhaven




msg:4386526
 8:12 pm on Nov 13, 2011 (gmt 0)

You might want to change:
RewriteCond %{REMOTE_ADDR} !^65\.(54|55)\.

to:
RewriteCond %{REMOTE_ADDR} !^65\.(52|54|55)\.

I've seen them come in through that range and it maps back to them. :)

incrediBILL




msg:4386919
 10:18 pm on Nov 14, 2011 (gmt 0)

I had an attempted drive by scraping from those ranges this morning.

Got hit requesting over 500 pages by about 45 different MS IPs, each asking for around 15-20 page each, all using this UA:

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)"

What in the hell is going on over there?

... and it's still ongoing, more page requests being made as I type this...

Pfui




msg:4386966
 12:46 am on Nov 15, 2011 (gmt 0)

1.) Are all the IPs bare/no rDNS, all .search.msn.com, or a mix?

2.) Are all the hits to public files? I ask because of:

msnbot-65-52-109-66.search.msn.com
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

21:45:31 /robots.txt
21:46:21 /access_logs/

That robots.txt-ignoring, literally-waaay-outta-line, never-public URI is why neither 65.52. nor bingbot have carte blanche access. (Took me a while to recall why, @motorhaven:)

motorhaven




msg:4387006
 3:54 am on Nov 15, 2011 (gmt 0)

Everything I've gotten (thousand of hits, its a forum with a very high page count) shows the 65.52 is legit for the crawlers.

Out of thousands of hits from the 65.62.#*$!.#*$! range the vast majority identify as a Microsoft crawler. These all fetch robots.txt and obey it.

The IPs below come in looking like either a user, or a stealth checker, but are rare in nature, hit a few pages and then leave. No search engine UA with these.
65.52.6.105
65.52.7.30
65.52.7.177
65.52.21.72
65.52.33.140
65.52.33.130
These do not fetch robots.txt but do obey it.

incrediBILL




msg:4387030
 5:39 am on Nov 15, 2011 (gmt 0)

The IPs hitting my site all rDNS to .search.msn.com with 891 hits so far today and counting

Also noticed a couple of different UAs from the same ranges, valid rDNS, also asking for hundreds of pages.

OK, going to do an MS lock down, screw this...

Pfui




msg:4387155
 2:26 pm on Nov 15, 2011 (gmt 0)

@motorhaven: For more about the 65.52s and 207.46s, see: MSN's Stealth Missions [webmasterworld.com...]

motorhaven




msg:4387193
 4:10 pm on Nov 15, 2011 (gmt 0)

Thanks. I let in their stealth bots. There's nothing to hide on my site, the search engine crawlers get the same thing the users get.

HenryUK




msg:4388483
 10:14 am on Nov 18, 2011 (gmt 0)

Since about Nov 8th I'm getting hit 50,000 - 80,000 times per day by a bot with the useragent

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)

Selection of IP addresses:


207.46.12.75
207.46.204.162
207.46.204.164
207.46.12.73
157.55.112.203
157.55.112.208

Not identified as a bot. And it is executing Javascript, which is using up a lot of resource and buggering up our stats.

Are other users just blocking via user-agent?

wilderness




msg:4388487
 10:31 am on Nov 18, 2011 (gmt 0)

try this

# ends with Gecko and from IP ranges
RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.(12|204)\.
RewriteCond %{REMOTE_ADDR} ^157\.55\.20[38]\.
RewriteRule .* - [F]

keyplyr




msg:4388515
 11:57 am on Nov 18, 2011 (gmt 0)

Actually wilderness, ya left out a C range:

RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.(12|204)\.
RewriteCond %{REMOTE_ADDR} ^157\.55\.112\.20[38]\.
RewriteRule .* - [F]



But if blocking with this method, IMO it would be more pro-effective to use complete ranges:

207.46.0.0 - 207.46.255.255
157.54.0.0 - 157.60.255.255

RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.
RewriteCond %{REMOTE_ADDR} ^157\.[56][0-9]\.
RewriteRule .* - [F]

HenryUK




msg:4388539
 1:41 pm on Nov 18, 2011 (gmt 0)

Thanks for the suggestions guys.

g1smd




msg:4388540
 1:45 pm on Nov 18, 2011 (gmt 0)

Syntax error! The code doesn't do what you want.

^157\.55\.112\.20[38]\. blocks 157.55.112.203 and 157.55.112.208

Actually, it doesn't even do that because of the trailing period.

Pfui




msg:4388569
 3:32 pm on Nov 18, 2011 (gmt 0)

Whatever this is --

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)

-- that's running amok on IncrediBill's and HenryUK's sites from MSN, also has an atypical + in the "Kit/" version spot in addition to missing the post-"Gecko)" version number. So the preceding example --

RewriteCond %{HTTP_USER_AGENT} Gecko\)$ [NC]
(Note: I escaped the closing paren)

-- or a variation on a theme --

RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*Gecko\)$ [NC]

-- should suffice, imo, regardless of REMOTE_ADDR. I don't care where it comes from, it's not getting in.

wilderness




msg:4388609
 6:03 pm on Nov 18, 2011 (gmt 0)

That's what I get for being awake in the wee hours.

Line Should read:

RewriteCond %{REMOTE_ADDR} ^157\.55\.112\.20[38]$

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved