homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How do I disallow bots from my Wordpress year Archive Folders
I guess this is a common question but I can't find an answer

 1:35 am on Apr 16, 2007 (gmt 0)

How do I disallow bots from spidering my archive folders in Wordpress?

Here is how my URL's are structured:


I tried testing this in Webmaster Central, but it returned back with an allow status:

User-agent: *
Disallow: /*/2007/$



 7:10 pm on Apr 17, 2007 (gmt 0)

You can make a change in your header file or make a plugin.

To do this in your header insert this somewhere in the head tags. Before or after the Title tag is fine.

if (is_archive()){
echo "<meta name=\"robots\" content=\"noindex,follow\">\n";

This will put noindex on Category, Author and Date based pages. That may be too much. Here [codex.wordpress.org] is a page that shows the different is_ codes you can try.


 5:46 pm on May 3, 2007 (gmt 0)


Question...This accomplished the same thing as the [seologs.com...] plug-in, correct?

I used the following code to archive only the index, page, posts:

<?php if(is_home() ¦¦ is_single() ¦¦ is_page()){
echo "<meta name=\"robots\" content=\"index,follow\">";
} else {
echo "<meta name=\"robots\" content=\"noindex,follow\">";

which seemed to work fine, then I found the plug-in and that seems to address the issue as well. Your thought on one vs. the other?


 5:26 pm on May 5, 2007 (gmt 0)

Yes it is very close to that plugin. I made comments on that site about my fix. I also blogged about it. I would have linked to it but thought we could not do that here.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved