Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

MSN - Robots.txt



1:36 am on Feb 27, 2005 (gmt 0)

10+ Year Member

msn bot seems to ignore the robots.txt anyone have any ideas how to control it's crawl.


2:25 am on Feb 27, 2005 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

I've never had a bit of trouble with msnbot, at least not since msnbot/0.11, the second released version. We haven't had many problem reports in the robots.txt forum [webmasterworld.com], as there would be if the robot had any major problems.

Be sure you are using the correct user-agent string, and validate your robots.txt [searchengineworld.com] file.

Here's a sample with the user-agent string I use:

User-agent: msnbot
Disallow: /cgi-bin


Featured Threads

Hot Threads This Week

Hot Threads This Month