Welcome to WebmasterWorld Guest from 184.108.40.206 , register , free tools , login , search , subscribe , help , library , announcements , recent posts , open posts Subscribe and Support WebmasterWorld
MSN - Robots.txt shopgal msg:1535173 1:36 am on Feb 27, 2005 (gmt 0) msn bot seems to ignore the robots.txt anyone have any ideas how to control it's crawl.
jdMorgan msg:1535174 2:25 am on Feb 27, 2005 (gmt 0)
I've never had a bit of trouble with msnbot, at least not since msnbot/0.11, the second released version. We haven't had many problem reports in the robots.txt forum [ webmasterworld.com], as there would be if the robot had any major problems.
Be sure you are using the correct user-agent string, and
validate your robots.txt [ searchengineworld.com] file.
Here's a sample with the user-agent string I use:
User-agent: msnbot Disallow: /cgi-bin Jim