Microsoft Spider poking around?

seems to be behaving itself

[WitchLars]

Is anyone else seeing activity from these IP's?

131.107.65.225 - (blank UA)
131.107.137.47 - Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)
131.107.151.157 - (blank UA)
131.107.151.202 - (blank UA)

Only 131.107.137.47 asked for robots.txt, but all four have displayed non-human patterns and have obeyed robots.txt. Whois reports that they belong to Microsoft.

I did find a few older threads regarding activity from 131.107.3.* (BunnySlippers and such), but this looks new.

-Lars

[wilderness]

131.107.65.
gets 403's.
No poking around with blank UA'a

[fiestagirl]

Good catch, I was just getting a little suspicious about that myself. No ua, no referrer. starting on 3/13.
131.107.65.225
131.107.151.199
131.107.151.157

Golly, why would microsoft be sneaking around spidering without identifying themselves, I wonder?
Yeah, wilderness, I've been toying with denying the no ua, no referrer types myself.

BTW, Bunnyslippers has been coming around with:
131.107.170.*

[carfac]

Yeah, wilderness, I've been toying with denying the no ua, no referrer types myself.

I have, too. Actually, I did do it. But one site I have- a very popular one- was getting a lot of innocent users banned, so I stopped it on that one site.

I did it in httpd.conf. Anyone want the code?

I did it like this (note I did it after an exculsion for requests of robots.txt... if you are blocking things like refferer = 'ieaa.org, that should be done BEFORE the robots.txt exclusion)

RewriteRule /robots.txt$ - [NC,L]
RewriteCond %{HTTP_REFERER} ^-?$[NC]
RewriteCond %{HTTP_USER_AGENT}^-?$[NC]
RewriteRule .*-[F,L]
RewriteCond %{HTTP_REFERER}NULL[NC]
RewriteCond %{HTTP_USER_AGENT}NULL[NC]
RewriteRule .*-[F,L]

I am not sure the "NULL" and the no case directive are required, but I wanted to be sure I was doing it!

dave

[fiestagirl]

Just a heads up. It seems that MS noticed they were being banned for no ua/referrer and added this fake on 4/2:

Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.2;+.NET+CLR+1.1.4322)

[jrobbio]

Maybe they are serious about setting their own search engine.

[bull]

216.205.150.91 - - [07/Apr/2003:10:43:22 +0200] "GET /odp/bla/ HTTP/1.0" 200 2200 www.me.net "-" "Mozilla/4.0+(compatible;)" "-"

Note the "+" after Mozilla/4.0, this why I post it here (think they may be related). Obviously this is a fake. 216.205.150.91 = 91-216.205.150.dellhost.com, which runs also Apache.