homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

blocking dynamic parts of forum

 4:43 am on Apr 25, 2011 (gmt 0)

...is there a way to block in the robots.txt the dynamic url of


but allow


any help would be appreciated...




 4:58 am on Apr 25, 2011 (gmt 0)

It's better IMO to use the meta ROBOTS NOINDEX for those pages because only Google non-standard robots.txt enhancements is flexible enough to what you want leaving the other SEs still to be dealt with.


 5:22 am on Apr 25, 2011 (gmt 0)

thanks incrediBill,

i dont think i can use robots NOINDEX for those pages as they are automatically generate by my forum software, but i going to look into their FAQ to see if that is possible...


 8:58 am on Apr 25, 2011 (gmt 0)

You would have to conditionally include it per page, a programming change, very unlikely it's already supported but something they should support based on Google's stance on thin/duplicate content so maybe they'll implement it for you, can't hurt to ask.


 9:16 am on Apr 25, 2011 (gmt 0)

thanks again,

i will get into contact with the forum than,
last question regarding this, could i at least block for now
(for some reason i cant post that url it breaks into gibberish, here a screenshot of the url)


I wonder if i can block at least the ?app=core part


 7:59 pm on May 2, 2011 (gmt 0)


just to let you know, i tried the lesser problem for now and settled blocking the following;

Disallow: /forum/index.php?app=core
Disallow: /forum/index.php?app=members
Disallow: /forum/index.php?autocom
Disallow: /forum/index.php?s=

within 4 days about 5.000 pages were blocked and as a nice side effect the visitor level is almost back to pre panda level...


 5:51 pm on May 10, 2011 (gmt 0)

hi viggen,

i have a somewhat similar issue. I want to blog user profiles for a forum where the extension looks like such:


how can i block all of those profiles using robots.txt?

thanks in advance.


 6:42 pm on May 10, 2011 (gmt 0)

...i would assume

Disallow: /forum/profile.php?

should do the trick, if you have Google WMT you can test if it does block it correctly...

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved