Forum Moderators: phranque
My site 360 pages are existed and all the pages are indexed in all the major search engines(Google,Yahoo & MSN), but in that 260 pages are duplicate pages, so now I have placed all duplicate pages in robots.txt file not to crawl.
My question is: when I placed all 260 pages in robots.txt file will search engines drop my 260 pages from their index or just they wont crawl those pages
Will it decreases duplicate content percentage?
Waiting for the reply
Search engines will no longer crawl the pages and you'll likely see them gradually drop out of the index. Whether or not robots exclusion is a good approach depends on the particular content you have.
Personally, I view robots exclusion as more of a preventative measure than a fix. If, for instance, you had accidentally duplicated all of your news articles in both /news/ and /news-articles/ you would be better off deciding on the preferred location and (permanently) redirecting the articles over there.
