homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How to avoid double indexing

5+ Year Member

Msg#: 3491283 posted 10:10 am on Oct 30, 2007 (gmt 0)

Hi everbody

I have 2 domains and my website can be accessed with both addresses and www & non-www versions.

Now I only want www.domain2.com to be indexed by SE, because it has the best PR.

So if I create a robot.txt file with following content, will that work?

User-agent: WebCrawler
Disallow: [domain1.com...]
Disallow: [domain1.com...]
Disallow: [domain2.com...]

I tried redirets and mod rewrited before, but I don't really care if people can access the site differently... ( only care about SE indexing )

Hope somebody can help me with this

2 everybody



WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 3491283 posted 12:46 pm on Oct 30, 2007 (gmt 0)

The format you show above is not supported by robots.txt. Records in robots.txt specify only local URL-paths, and including the protocol and domain is not supported.

The correct solution to this problem is to use a 301-Moved Permanently redirect to redirect all non-canonical domain variations to the single canonical domain. This is a popular subject, and searches on WebmasterWorld for "domain canonicalization," "canonical domain," and "www non-www domain" will turn up hundreds of threads with discussion and code examples.

Leaving your site as it is now will result in PageRank and link-popularity being 'split' across the multiple domain variants -- in effect, making your site compete against itself in the search results. Attempts to promote more than one domain may even result in search engine duplicate-content penalties -- if the promotion is too heavy or the domains too numerous.



5+ Year Member

Msg#: 3491283 posted 4:43 pm on Nov 13, 2007 (gmt 0)

or you can edit your meta tags and not index the pages .
although that is the long way around.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved