Forum Moderators: Robert Charlton & goodroi
I know of a site that has the same exact content on www,example.com and also blahblah.example.com. However, if you go to blahblah.example.com/robots.txt, it says:
User-agent: *
Disallow: /
So my question - How can they get away with that? Does Googlebot really just ignore duplicate content violations (and potentially others) with a simple disallow in robots.txt? Could I have www.example.com, www.example.net, and www.example.org all showing the same exact content without any penalty to example.com assuming I disallow googlebot in robots.txt on www.example.net + www.example.org?