Page is a not externally linkable
Mokita - 3:16 am on Dec 29, 2007 (gmt 0)
See this thread for more info: Trying to restrict individual bots using robots.txt is not cloaking.
Marcia - I think Jim is referring to MSNbot failing to follow individual directions in robots.txt for its multiple bots - like msnbot-media, msnbot-news, msnbot-products etc. If you try to disallow those from crawling your site, it also has the effect of disallowing the generic msnbot and your site disappears from their index.
[webmasterworld.com...]