shaunm - 9:03 am on Jul 25, 2012 (gmt 0)
I never heard of using 'noindex' within a sitemap. What is the difference between the 'disallow' and 'noindex' parameters?
Suppose I want to block a single URL in my website not to be crawled and indexed, which is the best way?
I just came across a website which uses both in their robots.txt file to make sure that their pages are not indexed in the search results.
I would very much appreciate your help guys. Thanks a lot!