Welcome to WebmasterWorld Guest from 54.162.226.212

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Microsoft bot 157 ranges updated

DNS for IP ranges 157.55 and 157.56 updated

     
11:05 pm on Nov 7, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



I've been keeping a desultory eye on the MS 157 ranges for some time. Not sure when it actually happened - probably within the past few weeks - but MS seem to have taken notice of complaints - or just finally got their act together.

I had a handful of IPs tagged as msnbot in my database from a previous DNS scan some time back. The new ranges/sub-ranges (about 100) extend this range but also, in some cases, extend or reduce previously logged ranges. The list I now have, taken from DNS scans of the MS DNS servers, is included below. I think the list is accurate but I cannot guarantee it.

I may have missed a few breaks in the actual bot list IPs (eg there may be a gap in a sequence I missed) but it is essentially correct.

------------------------------------------
157.55.2.138 - 157.55.2.182
157.55.7.138 - 157.55.7.182
157.55.10.202 - 157.55.10.246
157.55.11.138 - 157.55.11.182
157.55.11.200 - 157.55.11.244
157.55.13.10 - 157.55.13.116
157.55.13.138 - 157.55.13.182
157.55.16.9 - 157.55.16.99
157.55.16.140 - 157.55.16.231
157.55.17.73 - 157.55.17.117
157.55.17.137 - 157.55.17.228
157.55.18.9 - 157.55.18.53
157.55.18.74 - 157.55.18.119
157.55.18.138 - 157.55.18.183
157.55.18.202 - 157.55.18.251
157.55.19.10 - 157.55.19.55
157.55.19.74 - 157.55.19.119
157.55.19.138 - 157.55.19.183
157.55.19.202 - 157.55.19.251
157.55.20.10 - 157.55.20.55
157.55.20.74 - 157.55.20.183
157.55.20.202 - 157.55.20.247
157.55.21.10 - 157.55.21.55
157.55.21.74 - 157.55.21.119
157.55.21.138 - 157.55.21.183
157.55.21.202 - 157.55.21.252
157.55.22.10 - 157.55.22.55
157.55.22.74 - 157.55.22.119
157.55.22.138 - 157.55.22.183
157.55.22.202 - 157.55.22.247
157.55.23.10 - 157.55.23.55
157.55.23.74 - 157.55.23.183
157.55.36.139 - 157.55.36.228
157.55.37.10 - 157.55.37.103
157.55.37.139 - 157.55.37.228
157.55.38.10 - 157.55.38.99
157.55.38.139 - 157.55.38.236
157.55.39.10 - 157.55.39.99
157.55.39.139 - 157.55.39.228
157.55.48.7 - 157.55.48.51
157.55.48.71 - 157.55.48.115
157.55.48.135 - 157.55.48.243
157.55.50.7 - 157.55.50.51
157.55.50.71 - 157.55.50.115
157.55.98.8 - 157.55.98.52
157.55.98.74 - 157.55.98.118
157.55.99.10 - 157.55.99.54
157.55.99.199 - 157.55.99.243
157.55.100.10 - 157.55.100.54
157.55.102.7 - 157.55.102.51
157.55.103.135 - 157.55.103.179
157.55.103.199 - 157.55.103.243
157.55.106.10 - 157.55.106.13
157.55.106.74 - 157.55.106.118
157.55.106.138 - 157.55.106.182
157.55.106.202 - 157.55.106.246
157.55.107.10 - 157.55.107.102
157.55.107.138 - 157.55.107.182
157.55.107.202 - 157.55.107.246
157.55.108.10 - 157.55.108.118
157.55.108.138 - 157.55.108.182
157.55.108.202 - 157.55.108.246
157.55.109.10 - 157.55.109.118
157.55.109.138 - 157.55.109.182
157.55.109.202 - 157.55.109.246
157.55.110.10 - 157.55.110.115
157.55.110.135 - 157.55.110.179
157.55.110.202 - 157.55.110.246
157.55.111.74 - 157.55.111.118
157.55.111.138 - 157.55.111.139
157.55.111.202 - 157.55.111.246
157.55.112.199 - 157.55.112.243
157.55.114.10 - 157.55.114.54
157.55.114.202 - 157.55.114.241
157.55.116.8 - 157.55.116.97
157.55.118.74 - 157.55.118.113
157.55.118.138 - 157.55.118.177
157.55.154.135 - 157.55.154.224
------------------------------------------
157.56.0.10 - 157.56.0.99
157.56.0.139 - 157.56.0.228
157.56.1.10 - 157.56.1.99
157.56.1.135 - 157.56.1.184
157.56.2.10 - 157.56.2.99
157.56.2.185 - 157.56.2.228
157.56.3.7 - 157.56.3.94
157.56.3.135 - 157.56.3.222
157.56.4.7 - 157.56.4.94
157.56.4.135 - 157.56.4.222
157.56.5.7 - 157.56.5.94
157.56.5.135 - 157.56.5.222
157.56.16.135 - 157.56.16.178
157.56.16.199 - 157.56.16.242
157.56.17.7 - 157.56.17.50
157.56.17.71 - 157.56.17.114
157.56.17.135 - 157.56.17.178
157.56.17.199 - 157.56.17.242
157.56.18.7 - 157.56.18.50
157.56.18.135 - 157.56.18.222
157.56.80.7 - 157.56.80.51
157.56.80.71 - 157.56.80.115
------------------------------------------
11:58 pm on Nov 7, 2011 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



MS seem to have taken notice of complaints

Not sure if this is related but I've been complaining to Bing, msnbot, et al about errors these bots create at my server; sometimes as many as a couple hundred per day. I've exchanged numerous emails with them over the last few months.

Yesterday all those errors stopped.
1:47 am on Nov 8, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Could be it was that recent. I did a single check on a repeatedly failed IP on Saturday and that showed the update.

I complained to bingdude some time back, in another forum on WebmasterWorld, about the IP and UA problems and nudged again when he popped up there a few days ago so maybe that did some good. Who knows? :(
1:48 pm on Nov 8, 2011 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



I'm curious: What do you do with the 157. info? Do you only allow msnbot/bingbot from those ranges? Or keep track for your own interest? Or--?
5:16 pm on Nov 8, 2011 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



here's dstiles 156 & 157 Class C's (the lines are too many IMO to separate by Class D's) to regex:

157\.55\.([27]|[1[0136-9]|2[0-3]|3[6-9]|48|50|9[89]|10[0236-9]|11[012468]|154)\.

157\.56\.([0-5]|1[678]|80)\.
11:39 pm on Nov 8, 2011 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



Pfui - I hold the ranges in a MySQL database, wherein are all other blockable ranges, short-term auto-blocks etc. When I detect a bingbot UA I check it against the IP. If there is no match, no access. Which is why I'd run up a massive block list of 157 IPs until this list got added.

Reason for the method is historical as much as anything - my security system is a decade or more old, built when IIS had no proper regex let alone a decent htaccess capability.
12:12 am on Nov 9, 2011 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Thanks, dstiles. I, too, must juggle expediency with capability (the server's; my own:) Thus when it comes to the 157s et al, things get a bit-ham-fisted: msnbot and bingbot are only okay from:

RewriteCond %{REMOTE_HOST} !\.(bing|live|msn)\.com$
RewriteCond %{REMOTE_ADDR} !^65\.(54|55)\.
RewriteCond %{REMOTE_ADDR} !^157\.55\.
RewriteCond %{REMOTE_ADDR} !^207\.46\.

(The jury's still out on 157.56.)
8:12 pm on Nov 13, 2011 (gmt 0)

10+ Year Member



You might want to change:
RewriteCond %{REMOTE_ADDR} !^65\.(54|55)\.

to:
RewriteCond %{REMOTE_ADDR} !^65\.(52|54|55)\.

I've seen them come in through that range and it maps back to them. :)
10:18 pm on Nov 14, 2011 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I had an attempted drive by scraping from those ranges this morning.

Got hit requesting over 500 pages by about 45 different MS IPs, each asking for around 15-20 page each, all using this UA:

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)"

What in the hell is going on over there?

... and it's still ongoing, more page requests being made as I type this...
12:46 am on Nov 15, 2011 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



1.) Are all the IPs bare/no rDNS, all .search.msn.com, or a mix?

2.) Are all the hits to public files? I ask because of:

msnbot-65-52-109-66.search.msn.com
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

21:45:31 /robots.txt
21:46:21 /access_logs/

That robots.txt-ignoring, literally-waaay-outta-line, never-public URI is why neither 65.52. nor bingbot have carte blanche access. (Took me a while to recall why, @motorhaven:)
3:54 am on Nov 15, 2011 (gmt 0)

10+ Year Member



Everything I've gotten (thousand of hits, its a forum with a very high page count) shows the 65.52 is legit for the crawlers.

Out of thousands of hits from the 65.62.#*$!.#*$! range the vast majority identify as a Microsoft crawler. These all fetch robots.txt and obey it.

The IPs below come in looking like either a user, or a stealth checker, but are rare in nature, hit a few pages and then leave. No search engine UA with these.
65.52.6.105
65.52.7.30
65.52.7.177
65.52.21.72
65.52.33.140
65.52.33.130
These do not fetch robots.txt but do obey it.
5:39 am on Nov 15, 2011 (gmt 0)

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The IPs hitting my site all rDNS to .search.msn.com with 891 hits so far today and counting

Also noticed a couple of different UAs from the same ranges, valid rDNS, also asking for hundreds of pages.

OK, going to do an MS lock down, screw this...
2:26 pm on Nov 15, 2011 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



@motorhaven: For more about the 65.52s and 207.46s, see: MSN's Stealth Missions [webmasterworld.com...]
4:10 pm on Nov 15, 2011 (gmt 0)

10+ Year Member



Thanks. I let in their stealth bots. There's nothing to hide on my site, the search engine crawlers get the same thing the users get.
10:14 am on Nov 18, 2011 (gmt 0)

10+ Year Member



Since about Nov 8th I'm getting hit 50,000 - 80,000 times per day by a bot with the useragent

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)

Selection of IP addresses:


207.46.12.75
207.46.204.162
207.46.204.164
207.46.12.73
157.55.112.203
157.55.112.208

Not identified as a bot. And it is executing Javascript, which is using up a lot of resource and buggering up our stats.

Are other users just blocking via user-agent?
10:31 am on Nov 18, 2011 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



try this

# ends with Gecko and from IP ranges
RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.(12|204)\.
RewriteCond %{REMOTE_ADDR} ^157\.55\.20[38]\.
RewriteRule .* - [F]
11:57 am on Nov 18, 2011 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Actually wilderness, ya left out a C range:

RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.(12|204)\.
RewriteCond %{REMOTE_ADDR} ^157\.55\.112\.20[38]\.
RewriteRule .* - [F]



But if blocking with this method, IMO it would be more pro-effective to use complete ranges:

207.46.0.0 - 207.46.255.255
157.54.0.0 - 157.60.255.255

RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.
RewriteCond %{REMOTE_ADDR} ^157\.[56][0-9]\.
RewriteRule .* - [F]
1:41 pm on Nov 18, 2011 (gmt 0)

10+ Year Member



Thanks for the suggestions guys.
1:45 pm on Nov 18, 2011 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Syntax error! The code doesn't do what you want.

^157\.55\.112\.20[38]\.
blocks
157.55.112.203
and
157.55.112.208


Actually, it doesn't even do that because of the trailing period.
3:32 pm on Nov 18, 2011 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Whatever this is --

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)

-- that's running amok on IncrediBill's and HenryUK's sites from MSN, also has an atypical + in the "Kit/" version spot in addition to missing the post-"Gecko)" version number. So the preceding example --

RewriteCond %{HTTP_USER_AGENT} Gecko\)$ [NC]
(Note: I escaped the closing paren)

-- or a variation on a theme --

RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*Gecko\)$ [NC]

-- should suffice, imo, regardless of REMOTE_ADDR. I don't care where it comes from, it's not getting in.
6:03 pm on Nov 18, 2011 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



That's what I get for being awake in the wee hours.

Line Should read:

RewriteCond %{REMOTE_ADDR} ^157\.55\.112\.20[38]$
 

Featured Threads

Hot Threads This Week

Hot Threads This Month