Forum Moderators: open
So far, so good, robots.txt-wise. But it's not hit any of my embedded-media pages so I don't know what "media" it's looking for that msnbot doesn't already find, or MS's plans for the results.
Given those unknowns, and because msnbot is allowed more places on-site than not, I just specifically disallowed this variation (the last thing I need is people just hitting media files). We shall see if it heeds its own robots.txt designation:
User-agent: msnbot-media
Disallow: /
Access Log Examples:
tide525.microsoft.com - - [09/May/2006:07:49:02 -0700] "GET /robots.txt HTTP/1.1" 200 8382 "-"
"msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
tide525.microsoft.com - - [09/May/2006:07:49:05 -0700] "GET / HTTP/1.1" 200 14182 "-"
"msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
bl1sch2043804.phx.gbl - - [09/May/2006:16:43:06 -0700] "GET /robots.txt HTTP/1.0" 200 8382 "-"
"msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
bl1sch2043804.phx.gbl - - [09/May/2006:16:43:07 -0700] "GET / HTTP/1.0" 200 14147 "-"
"msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
[Multiple daily hits snipped]
bl1sch2043804.phx.gbl - - [16/May/2006:11:30:59 -0700] "GET /robots.txt HTTP/1.0" 200 8482 "-"
"msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
bl1sch2043804.phx.gbl - - [16/May/2006:11:31:03 -0700] "GET /dir/FILENAME.html HTTP/1.0" 200 52406 "-"
"msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)"
Is .gbl a valid top level domain? I don't think it is. [dnsstuff.com...]
That being said, I've definitely seen MSN bots with Microsoft IPs using a *.phx.gbl hostname. Am I missing something?
Hmm. I wonder if this elephant in our communal livingroom is a mailserver network? More importantly, I wonder what happens if we stop feeding it?
