Page is a not externally linkable
morags - 9:46 am on Oct 17, 2006 (gmt 0)
How do you prevent msnbot from indexing a page? This page has "<meta name="robots" content="noindex,nofollow">" in the head. ALL bots are also excluded from this page via robots.txt. And yet the page IS indexed and appears on search.live.com. Google and Yahoo ignore this page as expected. How do I prevent msnbot from indexing the page? Do I need to specifically mention msnbot in robots.txt? I am becoming very worried about possible duplicate content problems. Thanks.
Further to the thread below - but slightly different: