Here's an odd question - how to properly remove your site from a search engine.
I have two domains and while the content is not identical, it is similar enough and cross linked enough that a Yahoo rep once told me it was the reason I was being penalized. One of the domains was a private, paid access site which was cross linked to the sister sales site, so I subsequently added a robots.txt file and disallow meta tag to prevent the private site from being indexed.
Recently I complained that MSN was still indexing the site. I was contacted by MD, and he pointed out that my robots.txt file was dissallowing msnbot from reading the index page that contained the meta disallow tag that msnbot needs to see to stop indexing that page.
here's the existing robots.txt file:
# Robots.txt file
#
# Bans All Search Engines
# This is a members only website and need not be spidered.
# Disallow All
User-agent: *
Disallow: /
My question is, and msndude plase feel free to answer:
What exactly should I be doing to have this site, AND ONLY this site removed from MSN....and actually every search engine that considers two cross linked sites a no no....or is this not the proper approach?
The two domains were needed initially as a work management tool.
I eventually will be merging all the content into the one domain.