homepage Welcome to WebmasterWorld Guest from 50.16.36.153
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Accredited PayPal World Seller

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library : Charter : Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to avoid double indexing
mavi




msg:3491285		 10:10 am on Oct 30, 2007 (gmt 0)

Hi everbody

I have 2 domains and my website can be accessed with both addresses and www & non-www versions.

Now I only want www.domain2.com to be indexed by SE, because it has the best PR.

So if I create a robot.txt file with following content, will that work?

--------------------------------------------
User-agent: WebCrawler
Disallow: [domain1.com...]
Disallow: [domain1.com...]
Disallow: [domain2.com...]
--------------------------------------------

I tried redirets and mod rewrited before, but I don't really care if people can access the site differently... ( only care about SE indexing )

Hope somebody can help me with this

Reards
2 everybody
MAVI

 

jdMorgan




msg:3491377		 12:46 pm on Oct 30, 2007 (gmt 0)

The format you show above is not supported by robots.txt. Records in robots.txt specify only local URL-paths, and including the protocol and domain is not supported.

The correct solution to this problem is to use a 301-Moved Permanently redirect to redirect all non-canonical domain variations to the single canonical domain. This is a popular subject, and searches on WebmasterWorld for "domain canonicalization," "canonical domain," and "www non-www domain" will turn up hundreds of threads with discussion and code examples.

Leaving your site as it is now will result in PageRank and link-popularity being 'split' across the multiple domain variants -- in effect, making your site compete against itself in the search results. Attempts to promote more than one domain may even result in search engine duplicate-content penalties -- if the promotion is too heavy or the domains too numerous.

Jim

malcolmcroucher




msg:3503858		 4:43 pm on Nov 13, 2007 (gmt 0)

or you can edit your meta tags and not index the pages .
although that is the long way around.
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved