De-indexing a domain

will robots.txt be enough?

[bw3ttt]

I have a .co.uk site that is mirrored with a .com. The content is identical and I believe it is causing me to be penalized. I want Google to de-index the .com version. I have blocked all Google crawlers from crawling the .com version with robots.txt. Will this result in a complete de-indexing of the domain? Is there a faster way?

[BillyS]

You can remove the entire website via Google's Webmaster Tools. That will remove it from the index for 3 to 6 months then you can block Googlebot in robots.txt.

[Robert Charlton]

bw3ttt - You might want to take a look at these two threads regarding the issues involved with international domains with dupe content. Hosting and linking can be a big factor in English language dupes.

Duplicate Content on Localised Search
[webmasterworld.com...]

Linking a .ca to a .com with same content
[webmasterworld.com...]

De-indexing a .com is an extreme measure. I recommend checking out your hosting first, possibly moving one of the sites. I'd also check backlink sources, to make sure you're not de-indexing the site with best inbounds but the wrong hosting for the UK. And I'd also look into differentiating the content of the sites enough that they are localized for the US and the UK.

If you are going to dump the .com, and it does have inbound links, I'd think the most useful way of doing it would be to 301 redirect it to the .co.uk site... not to block it and throw away the link reputation it's built up.

[mjwalshe]

Why not just 301 the .com to the .co.uk via a .htacess file

[bw3ttt]

Too late.. I submitted a de-inclusion request and the .com mirror (which was essentially useless) is now completely gone.. I'm a little wary of doing anything at all that they might not like and there's something about redirects that bugs me. From their perspective there is nothing good about having a mirror with identical content wasting their hard drive space.

[bw3ttt]

oh and all IBLs point to the .co.uk so it's really no big loss..

[Robert Charlton]

I'm a little wary of doing anything at all that they might not like and there's something about redirects that bugs me. From their perspective there is nothing good about having a mirror with identical content wasting their hard drive space.

bw3ttt - Since you've already delisted the .com and it didn't have any inbound links anyway, the 301 would be pointless... but, just for the record, in case someone reads this thread down the road and is in a similar situation, 301s aren't browser redirects, as your comments above suggest.

301s are server side "permanent redirects," and when you 301 a site from domain1 to domain2, domain1 effectively ceases to exist. No content needs to be hosted.

The most efficient way to do a permanent redirect would be to address the DNS for domain1 to domain2's IP, and then, on the domain2 server, rewrite requests for domain1 so they call up domain2. You would need to continue your ownership of domain1, as well, of course, as domain2.

On Apache, you'd use mod_rewrite in a file called .htaccess. Not all hosting situations permit the use of mod_rewrite, but that's a whole other topic.

[tedster]

There's one reason that you might want to 301 redirect the .com domain: type-in traffic. Whether there's value here or not would depend on the actual domain name, and whether there are users who might assume .com when your active domain is a .co.uk.

Even after the domain's removal from the Google index, a 301 redirect could deliver some value for this situation.