MSN - Robots.txt

[shopgal]

msn bot seems to ignore the robots.txt anyone have any ideas how to control it's crawl.

[jdMorgan]

I've never had a bit of trouble with msnbot, at least not since msnbot/0.11, the second released version. We haven't had many problem reports in the robots.txt forum [webmasterworld.com], as there would be if the robot had any major problems.

Be sure you are using the correct user-agent string, and validate your robots.txt [searchengineworld.com] file.

Here's a sample with the user-agent string I use:

User-agent: msnbot
Disallow: /cgi-bin

Jim