Welcome to WebmasterWorld Guest from 54.166.37.177

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Microsoft bot 157 ranges updated

DNS for IP ranges 157.55 and 157.56 updated

     
11:05 pm on Nov 7, 2011 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3135
votes: 4


I've been keeping a desultory eye on the MS 157 ranges for some time. Not sure when it actually happened - probably within the past few weeks - but MS seem to have taken notice of complaints - or just finally got their act together.

I had a handful of IPs tagged as msnbot in my database from a previous DNS scan some time back. The new ranges/sub-ranges (about 100) extend this range but also, in some cases, extend or reduce previously logged ranges. The list I now have, taken from DNS scans of the MS DNS servers, is included below. I think the list is accurate but I cannot guarantee it.

I may have missed a few breaks in the actual bot list IPs (eg there may be a gap in a sequence I missed) but it is essentially correct.

------------------------------------------
157.55.2.138 - 157.55.2.182
157.55.7.138 - 157.55.7.182
157.55.10.202 - 157.55.10.246
157.55.11.138 - 157.55.11.182
157.55.11.200 - 157.55.11.244
157.55.13.10 - 157.55.13.116
157.55.13.138 - 157.55.13.182
157.55.16.9 - 157.55.16.99
157.55.16.140 - 157.55.16.231
157.55.17.73 - 157.55.17.117
157.55.17.137 - 157.55.17.228
157.55.18.9 - 157.55.18.53
157.55.18.74 - 157.55.18.119
157.55.18.138 - 157.55.18.183
157.55.18.202 - 157.55.18.251
157.55.19.10 - 157.55.19.55
157.55.19.74 - 157.55.19.119
157.55.19.138 - 157.55.19.183
157.55.19.202 - 157.55.19.251
157.55.20.10 - 157.55.20.55
157.55.20.74 - 157.55.20.183
157.55.20.202 - 157.55.20.247
157.55.21.10 - 157.55.21.55
157.55.21.74 - 157.55.21.119
157.55.21.138 - 157.55.21.183
157.55.21.202 - 157.55.21.252
157.55.22.10 - 157.55.22.55
157.55.22.74 - 157.55.22.119
157.55.22.138 - 157.55.22.183
157.55.22.202 - 157.55.22.247
157.55.23.10 - 157.55.23.55
157.55.23.74 - 157.55.23.183
157.55.36.139 - 157.55.36.228
157.55.37.10 - 157.55.37.103
157.55.37.139 - 157.55.37.228
157.55.38.10 - 157.55.38.99
157.55.38.139 - 157.55.38.236
157.55.39.10 - 157.55.39.99
157.55.39.139 - 157.55.39.228
157.55.48.7 - 157.55.48.51
157.55.48.71 - 157.55.48.115
157.55.48.135 - 157.55.48.243
157.55.50.7 - 157.55.50.51
157.55.50.71 - 157.55.50.115
157.55.98.8 - 157.55.98.52
157.55.98.74 - 157.55.98.118
157.55.99.10 - 157.55.99.54
157.55.99.199 - 157.55.99.243
157.55.100.10 - 157.55.100.54
157.55.102.7 - 157.55.102.51
157.55.103.135 - 157.55.103.179
157.55.103.199 - 157.55.103.243
157.55.106.10 - 157.55.106.13
157.55.106.74 - 157.55.106.118
157.55.106.138 - 157.55.106.182
157.55.106.202 - 157.55.106.246
157.55.107.10 - 157.55.107.102
157.55.107.138 - 157.55.107.182
157.55.107.202 - 157.55.107.246
157.55.108.10 - 157.55.108.118
157.55.108.138 - 157.55.108.182
157.55.108.202 - 157.55.108.246
157.55.109.10 - 157.55.109.118
157.55.109.138 - 157.55.109.182
157.55.109.202 - 157.55.109.246
157.55.110.10 - 157.55.110.115
157.55.110.135 - 157.55.110.179
157.55.110.202 - 157.55.110.246
157.55.111.74 - 157.55.111.118
157.55.111.138 - 157.55.111.139
157.55.111.202 - 157.55.111.246
157.55.112.199 - 157.55.112.243
157.55.114.10 - 157.55.114.54
157.55.114.202 - 157.55.114.241
157.55.116.8 - 157.55.116.97
157.55.118.74 - 157.55.118.113
157.55.118.138 - 157.55.118.177
157.55.154.135 - 157.55.154.224
------------------------------------------
157.56.0.10 - 157.56.0.99
157.56.0.139 - 157.56.0.228
157.56.1.10 - 157.56.1.99
157.56.1.135 - 157.56.1.184
157.56.2.10 - 157.56.2.99
157.56.2.185 - 157.56.2.228
157.56.3.7 - 157.56.3.94
157.56.3.135 - 157.56.3.222
157.56.4.7 - 157.56.4.94
157.56.4.135 - 157.56.4.222
157.56.5.7 - 157.56.5.94
157.56.5.135 - 157.56.5.222
157.56.16.135 - 157.56.16.178
157.56.16.199 - 157.56.16.242
157.56.17.7 - 157.56.17.50
157.56.17.71 - 157.56.17.114
157.56.17.135 - 157.56.17.178
157.56.17.199 - 157.56.17.242
157.56.18.7 - 157.56.18.50
157.56.18.135 - 157.56.18.222
157.56.80.7 - 157.56.80.51
157.56.80.71 - 157.56.80.115
------------------------------------------
11:58 pm on Nov 7, 2011 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:7040
votes: 183


MS seem to have taken notice of complaints

Not sure if this is related but I've been complaining to Bing, msnbot, et al about errors these bots create at my server; sometimes as many as a couple hundred per day. I've exchanged numerous emails with them over the last few months.

Yesterday all those errors stopped.
1:47 am on Nov 8, 2011 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3135
votes: 4


Could be it was that recent. I did a single check on a repeatedly failed IP on Saturday and that showed the update.

I complained to bingdude some time back, in another forum on WebmasterWorld, about the IP and UA problems and nudged again when he popped up there a few days ago so maybe that did some good. Who knows? :(
1:48 pm on Nov 8, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


I'm curious: What do you do with the 157. info? Do you only allow msnbot/bingbot from those ranges? Or keep track for your own interest? Or--?
5:16 pm on Nov 8, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


here's dstiles 156 & 157 Class C's (the lines are too many IMO to separate by Class D's) to regex:

157\.55\.([27]|[1[0136-9]|2[0-3]|3[6-9]|48|50|9[89]|10[0236-9]|11[012468]|154)\.

157\.56\.([0-5]|1[678]|80)\.
11:39 pm on Nov 8, 2011 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3135
votes: 4


Pfui - I hold the ranges in a MySQL database, wherein are all other blockable ranges, short-term auto-blocks etc. When I detect a bingbot UA I check it against the IP. If there is no match, no access. Which is why I'd run up a massive block list of 157 IPs until this list got added.

Reason for the method is historical as much as anything - my security system is a decade or more old, built when IIS had no proper regex let alone a decent htaccess capability.
12:12 am on Nov 9, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


Thanks, dstiles. I, too, must juggle expediency with capability (the server's; my own:) Thus when it comes to the 157s et al, things get a bit-ham-fisted: msnbot and bingbot are only okay from:

RewriteCond %{REMOTE_HOST} !\.(bing|live|msn)\.com$
RewriteCond %{REMOTE_ADDR} !^65\.(54|55)\.
RewriteCond %{REMOTE_ADDR} !^157\.55\.
RewriteCond %{REMOTE_ADDR} !^207\.46\.

(The jury's still out on 157.56.)
8:12 pm on Nov 13, 2011 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 10, 2004
posts: 399
votes: 13


You might want to change:
RewriteCond %{REMOTE_ADDR} !^65\.(54|55)\.

to:
RewriteCond %{REMOTE_ADDR} !^65\.(52|54|55)\.

I've seen them come in through that range and it maps back to them. :)
10:18 pm on Nov 14, 2011 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


I had an attempted drive by scraping from those ranges this morning.

Got hit requesting over 500 pages by about 45 different MS IPs, each asking for around 15-20 page each, all using this UA:

"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)"

What in the hell is going on over there?

... and it's still ongoing, more page requests being made as I type this...
12:46 am on Nov 15, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


1.) Are all the IPs bare/no rDNS, all .search.msn.com, or a mix?

2.) Are all the hits to public files? I ask because of:

msnbot-65-52-109-66.search.msn.com
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

21:45:31 /robots.txt
21:46:21 /access_logs/

That robots.txt-ignoring, literally-waaay-outta-line, never-public URI is why neither 65.52. nor bingbot have carte blanche access. (Took me a while to recall why, @motorhaven:)
3:54 am on Nov 15, 2011 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 10, 2004
posts: 399
votes: 13


Everything I've gotten (thousand of hits, its a forum with a very high page count) shows the 65.52 is legit for the crawlers.

Out of thousands of hits from the 65.62.#*$!.#*$! range the vast majority identify as a Microsoft crawler. These all fetch robots.txt and obey it.

The IPs below come in looking like either a user, or a stealth checker, but are rare in nature, hit a few pages and then leave. No search engine UA with these.
65.52.6.105
65.52.7.30
65.52.7.177
65.52.21.72
65.52.33.140
65.52.33.130
These do not fetch robots.txt but do obey it.
5:39 am on Nov 15, 2011 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


The IPs hitting my site all rDNS to .search.msn.com with 891 hits so far today and counting

Also noticed a couple of different UAs from the same ranges, valid rDNS, also asking for hundreds of pages.

OK, going to do an MS lock down, screw this...
2:26 pm on Nov 15, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


@motorhaven: For more about the 65.52s and 207.46s, see: MSN's Stealth Missions [webmasterworld.com...]
4:10 pm on Nov 15, 2011 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 10, 2004
posts: 399
votes: 13


Thanks. I let in their stealth bots. There's nothing to hide on my site, the search engine crawlers get the same thing the users get.
10:14 am on Nov 18, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
posts: 165
votes: 0


Since about Nov 8th I'm getting hit 50,000 - 80,000 times per day by a bot with the useragent

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)

Selection of IP addresses:


207.46.12.75
207.46.204.162
207.46.204.164
207.46.12.73
157.55.112.203
157.55.112.208

Not identified as a bot. And it is executing Javascript, which is using up a lot of resource and buggering up our stats.

Are other users just blocking via user-agent?
10:31 am on Nov 18, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


try this

# ends with Gecko and from IP ranges
RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.(12|204)\.
RewriteCond %{REMOTE_ADDR} ^157\.55\.20[38]\.
RewriteRule .* - [F]
11:57 am on Nov 18, 2011 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:7040
votes: 183


Actually wilderness, ya left out a C range:

RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.(12|204)\.
RewriteCond %{REMOTE_ADDR} ^157\.55\.112\.20[38]\.
RewriteRule .* - [F]



But if blocking with this method, IMO it would be more pro-effective to use complete ranges:

207.46.0.0 - 207.46.255.255
157.54.0.0 - 157.60.255.255

RewriteCond %{HTTP_USER_AGENT} Gecko)$
RewriteCond %{REMOTE_ADDR} ^207\.46\.
RewriteCond %{REMOTE_ADDR} ^157\.[56][0-9]\.
RewriteRule .* - [F]
1:41 pm on Nov 18, 2011 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
posts: 165
votes: 0


Thanks for the suggestions guys.
1:45 pm on Nov 18, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Syntax error! The code doesn't do what you want.

^157\.55\.112\.20[38]\.
blocks
157.55.112.203
and
157.55.112.208


Actually, it doesn't even do that because of the trailing period.
3:32 pm on Nov 18, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2038
votes: 1


Whatever this is --

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko)

-- that's running amok on IncrediBill's and HenryUK's sites from MSN, also has an atypical + in the "Kit/" version spot in addition to missing the post-"Gecko)" version number. So the preceding example --

RewriteCond %{HTTP_USER_AGENT} Gecko\)$ [NC]
(Note: I escaped the closing paren)

-- or a variation on a theme --

RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*Gecko\)$ [NC]

-- should suffice, imo, regardless of REMOTE_ADDR. I don't care where it comes from, it's not getting in.
6:03 pm on Nov 18, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5459
votes: 3


That's what I get for being awake in the wee hours.

Line Should read:

RewriteCond %{REMOTE_ADDR} ^157\.55\.112\.20[38]$
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members