Welcome to WebmasterWorld Guest from 54.162.240.235

Forum Moderators: mack

Message Too Old, No Replies

MSN - Robots.txt

     
1:36 am on Feb 27, 2005 (gmt 0)

10+ Year Member



msn bot seems to ignore the robots.txt anyone have any ideas how to control it's crawl.
2:25 am on Feb 27, 2005 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I've never had a bit of trouble with msnbot, at least not since msnbot/0.11, the second released version. We haven't had many problem reports in the robots.txt forum [webmasterworld.com], as there would be if the robot had any major problems.

Be sure you are using the correct user-agent string, and validate your robots.txt [searchengineworld.com] file.

Here's a sample with the user-agent string I use:


User-agent: msnbot
Disallow: /cgi-bin

Jim
 

Featured Threads

Hot Threads This Week

Hot Threads This Month