Welcome to WebmasterWorld Guest from 22.214.171.124 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
MSN - Robots.txt shopgal
msn bot seems to ignore the robots.txt anyone have any ideas how to control it's crawl.
I've never had a bit of trouble with msnbot, at least not since msnbot/0.11, the second released version. We haven't had many problem reports in the robots.txt forum [ webmasterworld.com], as there would be if the robot had any major problems.
Be sure you are using the correct user-agent string, and
validate your robots.txt [ searchengineworld.com] file.
Here's a sample with the user-agent string I use:
User-agent: msnbot Disallow: /cgi-bin Jim