homepage Welcome to WebmasterWorld Guest from 54.163.72.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
blocking dynamic parts of forum
viggen




msg:4303492
 4:43 am on Apr 25, 2011 (gmt 0)

...is there a way to block in the robots.txt the dynamic url of

../forum/index.php?app=blog&module=actions&section=forward&do=sendentry&eid=195&blogid=22

but allow

../forum/index.php?app=blog&module=display&section=blog&blogid=31&showentry=2531

any help would be appreciated...

cheers
viggen

 

incrediBILL




msg:4303494
 4:58 am on Apr 25, 2011 (gmt 0)

It's better IMO to use the meta ROBOTS NOINDEX for those pages because only Google non-standard robots.txt enhancements is flexible enough to what you want leaving the other SEs still to be dealt with.

viggen




msg:4303498
 5:22 am on Apr 25, 2011 (gmt 0)

thanks incrediBill,

i dont think i can use robots NOINDEX for those pages as they are automatically generate by my forum software, but i going to look into their FAQ to see if that is possible...

incrediBILL




msg:4303545
 8:58 am on Apr 25, 2011 (gmt 0)

You would have to conditionally include it per page, a programming change, very unlikely it's already supported but something they should support based on Google's stance on thin/duplicate content so maybe they'll implement it for you, can't hurt to ask.

viggen




msg:4303549
 9:16 am on Apr 25, 2011 (gmt 0)

thanks again,

i will get into contact with the forum than,
last question regarding this, could i at least block for now
(for some reason i cant post that url it breaks into gibberish, here a screenshot of the url)

[img842.imageshack.us...]

I wonder if i can block at least the ?app=core part

viggen




msg:4307055
 7:59 pm on May 2, 2011 (gmt 0)

@incredibill

just to let you know, i tried the lesser problem for now and settled blocking the following;

Disallow: /forum/index.php?app=core
Disallow: /forum/index.php?app=members
Disallow: /forum/index.php?autocom
Disallow: /forum/index.php?s=

within 4 days about 5.000 pages were blocked and as a nice side effect the visitor level is almost back to pre panda level...

OrganicPop




msg:4310442
 5:51 pm on May 10, 2011 (gmt 0)

hi viggen,

i have a somewhat similar issue. I want to blog user profiles for a forum where the extension looks like such:

/forum/profile.php?id=227

how can i block all of those profiles using robots.txt?

thanks in advance.

viggen




msg:4310460
 6:42 pm on May 10, 2011 (gmt 0)

...i would assume

Disallow: /forum/profile.php?

should do the trick, if you have Google WMT you can test if it does block it correctly...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved