There've been numerous threads about the duplicate content problems and several related problems.
Problem 1: www and non-www are seen as two different sites but if you have a non-www robots.txt blocking Googlebot ... Google sees that as a block on both the non-www and the www pages. thread [webmasterworld.com]
Problem 2: Supplementals just don't go away. Matt keeps saying there's no problem but the reality is that you may be able to get a page to go out of supplemental for a while but it will pop straight back in again [webmasterworld.com].
Problem 3: Google removal tool - Apart from the fact that it's too easy... you state something in your robots.txt and competitors are rushing to use the Removal Tool to get you in trouble - what's the point of a tool that doesn't do what it says it will do?
Matt Cutts said in early October (in his moving to a new web host post) that they were doing something about 301 within the next "couple of months" or so. That's no promise, of course, and we have to work on the assumption that Google may never fix the broken bits.
I have some questions that I'm hoping some of you more experienced with the problem can help with. I appreciate that the removal tool is dangerous and you can knock your site out for 90/180 days. But, on some sites the problem is so bad that it really may be worth starting again.
1) Is using the removal tool to remove the whole site the same as using robots.txt to ban Google. If robots.txt is a low-strength medicine is that preferable i.e. ban googlebot, all google crawls stop, google stops showing your page, you re-allow Googlebot, it re-indexes, you've lost a few days traffic but have now got all your pages indexed and indexed correctly. (It can't be that simple, can it?)
2) What do you do if yoursite.com/index.htm is a duplicate of www.yoursite.com/index.htm. OK, you've got your 301 in place but how do you treat the removal of the index.htm page?
3) How do you get https pages removed? The removal tool doesn't do this. What other options are there?
4) If you use the removal tool and remove your whole site for a while... when your site comes back do you still benefit from all your backlinks? I'm assuming that removing your site from Google's index is not as serious as allowing your domain to expire :)