Microsoft bot 157 ranges updated

DNS for IP ranges 157.55 and 157.56 updated

[dstiles]

I've been keeping a desultory eye on the MS 157 ranges for some time. Not sure when it actually happened - probably within the past few weeks - but MS seem to have taken notice of complaints - or just finally got their act together.

I had a handful of IPs tagged as msnbot in my database from a previous DNS scan some time back. The new ranges/sub-ranges (about 100) extend this range but also, in some cases, extend or reduce previously logged ranges. The list I now have, taken from DNS scans of the MS DNS servers, is included below. I think the list is accurate but I cannot guarantee it.

I may have missed a few breaks in the actual bot list IPs (eg there may be a gap in a sequence I missed) but it is essentially correct.

------------------------------------------
157.55.2.138 - 157.55.2.182
157.55.7.138 - 157.55.7.182
157.55.10.202 - 157.55.10.246
157.55.11.138 - 157.55.11.182
157.55.11.200 - 157.55.11.244
157.55.13.10 - 157.55.13.116
157.55.13.138 - 157.55.13.182
157.55.16.9 - 157.55.16.99
157.55.16.140 - 157.55.16.231
157.55.17.73 - 157.55.17.117
157.55.17.137 - 157.55.17.228
157.55.18.9 - 157.55.18.53
157.55.18.74 - 157.55.18.119
157.55.18.138 - 157.55.18.183
157.55.18.202 - 157.55.18.251
157.55.19.10 - 157.55.19.55
157.55.19.74 - 157.55.19.119
157.55.19.138 - 157.55.19.183
157.55.19.202 - 157.55.19.251
157.55.20.10 - 157.55.20.55
157.55.20.74 - 157.55.20.183
157.55.20.202 - 157.55.20.247
157.55.21.10 - 157.55.21.55
157.55.21.74 - 157.55.21.119
157.55.21.138 - 157.55.21.183
157.55.21.202 - 157.55.21.252
157.55.22.10 - 157.55.22.55
157.55.22.74 - 157.55.22.119
157.55.22.138 - 157.55.22.183
157.55.22.202 - 157.55.22.247
157.55.23.10 - 157.55.23.55
157.55.23.74 - 157.55.23.183
157.55.36.139 - 157.55.36.228
157.55.37.10 - 157.55.37.103
157.55.37.139 - 157.55.37.228
157.55.38.10 - 157.55.38.99
157.55.38.139 - 157.55.38.236
157.55.39.10 - 157.55.39.99
157.55.39.139 - 157.55.39.228
157.55.48.7 - 157.55.48.51
157.55.48.71 - 157.55.48.115
157.55.48.135 - 157.55.48.243
157.55.50.7 - 157.55.50.51
157.55.50.71 - 157.55.50.115
157.55.98.8 - 157.55.98.52
157.55.98.74 - 157.55.98.118
157.55.99.10 - 157.55.99.54
157.55.99.199 - 157.55.99.243
157.55.100.10 - 157.55.100.54
157.55.102.7 - 157.55.102.51
157.55.103.135 - 157.55.103.179
157.55.103.199 - 157.55.103.243
157.55.106.10 - 157.55.106.13
157.55.106.74 - 157.55.106.118
157.55.106.138 - 157.55.106.182
157.55.106.202 - 157.55.106.246
157.55.107.10 - 157.55.107.102
157.55.107.138 - 157.55.107.182
157.55.107.202 - 157.55.107.246
157.55.108.10 - 157.55.108.118
157.55.108.138 - 157.55.108.182
157.55.108.202 - 157.55.108.246
157.55.109.10 - 157.55.109.118
157.55.109.138 - 157.55.109.182
157.55.109.202 - 157.55.109.246
157.55.110.10 - 157.55.110.115
157.55.110.135 - 157.55.110.179
157.55.110.202 - 157.55.110.246
157.55.111.74 - 157.55.111.118
157.55.111.138 - 157.55.111.139
157.55.111.202 - 157.55.111.246
157.55.112.199 - 157.55.112.243
157.55.114.10 - 157.55.114.54
157.55.114.202 - 157.55.114.241
157.55.116.8 - 157.55.116.97
157.55.118.74 - 157.55.118.113
157.55.118.138 - 157.55.118.177
157.55.154.135 - 157.55.154.224
------------------------------------------
157.56.0.10 - 157.56.0.99
157.56.0.139 - 157.56.0.228
157.56.1.10 - 157.56.1.99
157.56.1.135 - 157.56.1.184
157.56.2.10 - 157.56.2.99
157.56.2.185 - 157.56.2.228
157.56.3.7 - 157.56.3.94
157.56.3.135 - 157.56.3.222
157.56.4.7 - 157.56.4.94
157.56.4.135 - 157.56.4.222
157.56.5.7 - 157.56.5.94
157.56.5.135 - 157.56.5.222
157.56.16.135 - 157.56.16.178
157.56.16.199 - 157.56.16.242
157.56.17.7 - 157.56.17.50
157.56.17.71 - 157.56.17.114
157.56.17.135 - 157.56.17.178
157.56.17.199 - 157.56.17.242
157.56.18.7 - 157.56.18.50
157.56.18.135 - 157.56.18.222
157.56.80.7 - 157.56.80.51
157.56.80.71 - 157.56.80.115
------------------------------------------

[keyplyr]

MS seem to have taken notice of complaints

Not sure if this is related but I've been complaining to Bing, msnbot, et al about errors these bots create at my server; sometimes as many as a couple hundred per day. I've exchanged numerous emails with them over the last few months.

Yesterday all those errors stopped.

[dstiles]

Could be it was that recent. I did a single check on a repeatedly failed IP on Saturday and that showed the update.

I complained to bingdude some time back, in another forum on WebmasterWorld, about the IP and UA problems and nudged again when he popped up there a few days ago so maybe that did some good. Who knows? :(

[Pfui]

I'm curious: What do you do with the 157. info? Do you only allow msnbot/bingbot from those ranges? Or keep track for your own interest? Or--?

[wilderness]

here's dstiles 156 & 157 Class C's (the lines are too many IMO to separate by Class D's) to regex:

157\.55\.([27]|[1[0136-9]|2[0-3]|3[6-9]|48|50|9[89]|10[0236-9]|11[012468]|154)\.

157\.56\.([0-5]|1[678]|80)\.

[dstiles]

Pfui - I hold the ranges in a MySQL database, wherein are all other blockable ranges, short-term auto-blocks etc. When I detect a bingbot UA I check it against the IP. If there is no match, no access. Which is why I'd run up a massive block list of 157 IPs until this list got added.

Reason for the method is historical as much as anything - my security system is a decade or more old, built when IIS had no proper regex let alone a decent htaccess capability.

[Pfui]

Thanks, dstiles. I, too, must juggle expediency with capability (the server's; my own:) Thus when it comes to the 157s et al, things get a bit-ham-fisted: msnbot and bingbot are only okay from:

RewriteCond %{REMOTE_HOST} !\.(bing|live|msn)\.com$
RewriteCond %{REMOTE_ADDR} !^65\.(54|55)\.
RewriteCond %{REMOTE_ADDR} !^157\.55\.
RewriteCond %{REMOTE_ADDR} !^207\.46\.

(The jury's still out on 157.56.)

[motorhaven]

You might want to change:
RewriteCond %{REMOTE_ADDR} !^65\.(54|55)\.

to:
RewriteCond %{REMOTE_ADDR} !^65\.(52|54|55)\.

I've seen them come in through that range and it maps back to them. :)

[incrediBILL]

I had an attempted drive by scraping from those ranges this morning.

Got hit requesting over 500 pages by about 45 different MS IPs, each asking for around 15-20 page each, all using this UA:

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)"

What in the hell is going on over there?

... and it's still ongoing, more page requests being made as I type this...

[Pfui]

1.) Are all the IPs bare/no rDNS, all .search.msn.com, or a mix?

2.) Are all the hits to public files? I ask because of:

msnbot-65-52-109-66.search.msn.com
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

21:45:31 /robots.txt
21:46:21 /access_logs/

That robots.txt-ignoring, literally-waaay-outta-line, never-public URI is why neither 65.52. nor bingbot have carte blanche access. (Took me a while to recall why, @motorhaven:)

[motorhaven]

Everything I've gotten (thousand of hits, its a forum with a very high page count) shows the 65.52 is legit for the crawlers.

Out of thousands of hits from the 65.62.#*$!.#*$! range the vast majority identify as a Microsoft crawler. These all fetch robots.txt and obey it.

The IPs below come in looking like either a user, or a stealth checker, but are rare in nature, hit a few pages and then leave. No search engine UA with these.
65.52.6.105
65.52.7.30
65.52.7.177
65.52.21.72
65.52.33.140
65.52.33.130
These do not fetch robots.txt but do obey it.

[incrediBILL]

The IPs hitting my site all rDNS to .search.msn.com with 891 hits so far today and counting

Also noticed a couple of different UAs from the same ranges, valid rDNS, also asking for hundreds of pages.

OK, going to do an MS lock down, screw this...

[Pfui]

@motorhaven: For more about the 65.52s and 207.46s, see: MSN's Stealth Missions [webmasterworld.com...]

[motorhaven]

Thanks. I let in their stealth bots. There's nothing to hide on my site, the search engine crawlers get the same thing the users get.

[HenryUK]

Since about Nov 8th I'm getting hit 50,000 - 80,000 times per day by a bot with the useragent

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)

Selection of IP addresses:


207.46.12.75
207.46.204.162
207.46.204.164
207.46.12.73
157.55.112.203
157.55.112.208

Not identified as a bot. And it is executing Javascript, which is using up a lot of resource and buggering up our stats.

Are other users just blocking via user-agent?

[wilderness]

try this

# ends with Gecko and from IP ranges
RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.(12|204)\.
RewriteCond %{REMOTE_ADDR} ^157\.55\.20[38]\.
RewriteRule .* - [F]

[keyplyr]

Actually wilderness, ya left out a C range:

RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.(12|204)\.
RewriteCond %{REMOTE_ADDR} ^157\.55\.112\.20[38]\.
RewriteRule .* - [F]



But if blocking with this method, IMO it would be more pro-effective to use complete ranges:

207.46.0.0 - 207.46.255.255
157.54.0.0 - 157.60.255.255

RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.
RewriteCond %{REMOTE_ADDR} ^157\.[56][0-9]\.
RewriteRule .* - [F]

[HenryUK]

Thanks for the suggestions guys.

[g1smd]

Syntax error! The code doesn't do what you want.

^157\.55\.112\.20[38]\.

blocks

157.55.112.203

and

157.55.112.208

Actually, it doesn't even do that because of the trailing period.

[Pfui]

Whatever this is --

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)

-- that's running amok on IncrediBill's and HenryUK's sites from MSN, also has an atypical + in the "Kit/" version spot in addition to missing the post-"Gecko)" version number. So the preceding example --

RewriteCond %{HTTP_USER_AGENT} Gecko\)$ [NC]

(Note: I escaped the closing paren)

-- or a variation on a theme --

RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*Gecko\)$ [NC]

-- should suffice, imo, regardless of REMOTE_ADDR. I don't care where it comes from, it's not getting in.

[wilderness]

That's what I get for being awake in the wee hours.

Line Should read:

RewriteCond %{REMOTE_ADDR} ^157\.55\.112\.20[38]$