lucy24 - 9:00 am on Dec 24, 2012 (gmt 0)
Are there any more MS ranges that we've never heard of in our entire lives?
The only msnbot-media robots.txt request that fail are the failed headers checks
So that's where we differ. I let everyone get robots.txt, no matter how evil. But then, I don't have one of those fancy rewrites that shows a different file to every robot so they can't switch UAs and come back dressed up as Good Robot.