Forum Moderators: mack

Message Too Old, No Replies

Duplicate Domains - similar content

         

microlinx

12:06 pm on Jul 27, 2006 (gmt 0)

10+ Year Member



Here's an odd question - how to properly remove your site from a search engine.
I have two domains and while the content is not identical, it is similar enough and cross linked enough that a Yahoo rep once told me it was the reason I was being penalized. One of the domains was a private, paid access site which was cross linked to the sister sales site, so I subsequently added a robots.txt file and disallow meta tag to prevent the private site from being indexed.
Recently I complained that MSN was still indexing the site. I was contacted by MD, and he pointed out that my robots.txt file was dissallowing msnbot from reading the index page that contained the meta disallow tag that msnbot needs to see to stop indexing that page.

here's the existing robots.txt file:
# Robots.txt file
#
# Bans All Search Engines
# This is a members only website and need not be spidered.
# Disallow All

User-agent: *
Disallow: /

My question is, and msndude plase feel free to answer:
What exactly should I be doing to have this site, AND ONLY this site removed from MSN....and actually every search engine that considers two cross linked sites a no no....or is this not the proper approach?

The two domains were needed initially as a work management tool.
I eventually will be merging all the content into the one domain.

Erku

11:17 pm on Jul 27, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Since we are talking about duplicate content msndude I would like to ask about Print This Page feature in our pages.

Presently I use
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Is this suggested for Print Pages, or not necessary?

Thank you.

piskie

11:26 pm on Jul 27, 2006 (gmt 0)

10+ Year Member



The best way to safeguard duplicate content in a members only site is to make it Password access only.

msndude

6:03 pm on Jul 28, 2006 (gmt 0)

10+ Year Member



For the specific links from your public site to your private site, why not use rel="nofollow"? That effectively tells us to act like those links are just plain text. Then we won't see any connection between the two sites -- which I think is what you intend.