homepage Welcome to WebmasterWorld Guest from 23.22.128.96
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How can I block these kinds of urls?
starchild




msg:4346153
 11:44 am on Aug 1, 2011 (gmt 0)

Hi,

My forum based site is getting hit hard by google duplicate content filters. I have pages such as the two examples below being indexed.

/my-forum/discussion/416/?Focus=6222
/my-forum/discussion/416/discussion-name/

How can I block pages such as /my-forum/discussion/*/?Focus*

Is that valid? If not how can I fix it please?

 

robzilla




msg:4346157
 12:12 pm on Aug 1, 2011 (gmt 0)

I would recommend implementing the canonical tag [googlewebmastercentral.blogspot.com], the effect being similar to a 301 redirect, over the blocking of pages via robots.txt.

phranque




msg:4346448
 1:45 am on Aug 2, 2011 (gmt 0)

if those urls return the same content and assuming /my-forum/discussion/416/discussion-name/ is the canonical url the proper technical solution would be to respond to the request for /my-forum/discussion/416/?Focus=6222 with a 301 redirect to the canonical url.
if you use robots.txt to exclude crawling of those urls they may appear in search results as url-only snippets, possibly with a title based on the anchor text of the inbound link.
also note that a 301 redirect is incontrovertible while rel canonical is a "suggestion" not a directive.
About rel="canonical":
http://www.google.com/support/webmasters/bin/answer.py?answer=139394 [google.com]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved