homepage Welcome to WebmasterWorld Guest from 54.198.42.105
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
MSN - Robots.txt
shopgal




msg:1535173
 1:36 am on Feb 27, 2005 (gmt 0)

msn bot seems to ignore the robots.txt anyone have any ideas how to control it's crawl.

 

jdMorgan




msg:1535174
 2:25 am on Feb 27, 2005 (gmt 0)

I've never had a bit of trouble with msnbot, at least not since msnbot/0.11, the second released version. We haven't had many problem reports in the robots.txt forum [webmasterworld.com], as there would be if the robot had any major problems.

Be sure you are using the correct user-agent string, and validate your robots.txt [searchengineworld.com] file.

Here's a sample with the user-agent string I use:

User-agent: msnbot
Disallow: /cgi-bin

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved