Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

MSN - Robots.txt

1:36 am on Feb 27, 2005 (gmt 0)

New User

10+ Year Member

joined:Nov 18, 2003
votes: 0

msn bot seems to ignore the robots.txt anyone have any ideas how to control it's crawl.
2:25 am on Feb 27, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0

I've never had a bit of trouble with msnbot, at least not since msnbot/0.11, the second released version. We haven't had many problem reports in the robots.txt forum [webmasterworld.com], as there would be if the robot had any major problems.

Be sure you are using the correct user-agent string, and validate your robots.txt [searchengineworld.com] file.

Here's a sample with the user-agent string I use:

User-agent: msnbot
Disallow: /cgi-bin