homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How to stop google from indexing feeds
google indexing feeds

Msg#: 4658534 posted 8:16 am on Mar 30, 2014 (gmt 0)

I am using wordpress and I think google have somewhat penalty me for duplicate contents because of my feeds.

the type of feeds that is indexed are


I read about blocking it using robots.txt but that will only block it, the links would still be indexed.

I can't find any way to stop google from indexing. Any ideas?


Hemanth Malli

Msg#: 4658534 posted 6:57 am on Jun 11, 2014 (gmt 0)

You are using the same keywords but you have to change the related categories with the specific keywords. Then google can never indexed you.


WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 4658534 posted 12:22 pm on Jun 11, 2014 (gmt 0)

welcome to WebmasterWorld, bearbrick!

you could use the X-Robots-Tag HTTP header:
http://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag [developers.google.com]


WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

Msg#: 4658534 posted 3:53 pm on Jun 11, 2014 (gmt 0)

Hi bearbrick, I think if you had asked this same question in the WordPress Forum you would have hard about various ways to control what gets into your sitemaps. The /tags/ directory is a virtual directory that does not really exist for example, it just gives people a different way to find your pages or posts listed by the tags you added. If /tags/ and /categories/ are in your sitemaps, Google will index those pages and view them as duplicate content.

Depending on your skills there are a number of ways such as the X-Robots tag that can send a message with particular URLs that the URL is not to be indexed. Many WP users resort to a plugin that lets them control what URLs get indexed. Overall, you want to have your Posts and Pages in your sitemap and not list your content any other way just because it is the same content, shown a different way. Some of these plugins also help with canonical links for the duplicate pages so even if they are found via links, it will not be seen as duplicate. I use Yoast's SEO plugin for that but it is not for everyone. You need to read how to set it up properly before using it.


Msg#: 4658534 posted 10:15 am on Oct 11, 2014 (gmt 0)

Just add these following code in your robots.txt file to stop indexing your blog feed.

Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /feed/

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved