Welcome to WebmasterWorld Guest from 22.214.171.124
Today MSN bot directly hit my php version of robots.txt (robots.#*$!.php) with a get and then immediately hit robots.txt with a get. No other bots/requests for robots.txt have ever tried/or succeeded in accessing the php version. While it wouldn’t be impossible to guess the #*$! it isn’t all that obvious and I have to believe the MSN bot knew what it was looking for.
I asked for a review of my .htaccess rewrite logic over in the apache forum. The problem was the order of my rewrites. I did the internal rewrite for robots.txt to robots.$!@#.php first which worked except,, later I did an external (304) rewrite appending www which exposed my internal rewrite. Most (all other) bots that I've logged went after robots.txt with a www. so I never saw the exposing of my internal rewrite. Thanks to Jim all fixed now.
So, msnbot must try to get robots.txt using a url with both the www. and without the www. Maybe they've learned that the technique can yield results.