homepage Welcome to WebmasterWorld Guest from 54.227.41.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How do I disallow bots from my Wordpress year Archive Folders
I guess this is a common question but I can't find an answer
Drum




msg:3312022
 1:35 am on Apr 16, 2007 (gmt 0)

How do I disallow bots from spidering my archive folders in Wordpress?

Here is how my URL's are structured:

http://www.example.com/blog/index.php/2007/04/

I tried testing this in Webmaster Central, but it returned back with an allow status:

User-agent: *
Disallow: /*/2007/$

 

ogletree




msg:3313787
 7:10 pm on Apr 17, 2007 (gmt 0)

You can make a change in your header file or make a plugin.

To do this in your header insert this somewhere in the head tags. Before or after the Title tag is fine.

<?php
if (is_archive()){
echo "<meta name=\"robots\" content=\"noindex,follow\">\n";
}
?>

This will put noindex on Category, Author and Date based pages. That may be too much. Here [codex.wordpress.org] is a page that shows the different is_ codes you can try.

chazeo




msg:3330020
 5:46 pm on May 3, 2007 (gmt 0)

Ogletree,

Question...This accomplished the same thing as the [seologs.com...] plug-in, correct?

I used the following code to archive only the index, page, posts:

<?php if(is_home() ¦¦ is_single() ¦¦ is_page()){
echo "<meta name=\"robots\" content=\"index,follow\">";
} else {
echo "<meta name=\"robots\" content=\"noindex,follow\">";
}?>

which seemed to work fine, then I found the plug-in and that seems to address the issue as well. Your thought on one vs. the other?

ogletree




msg:3331731
 5:26 pm on May 5, 2007 (gmt 0)

Yes it is very close to that plugin. I made comments on that site about my fix. I also blogged about it. I would have linked to it but thought we could not do that here.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved