Forum Moderators: open
How about uploading the new content onto the mirrors and robots.txt ( never dome that before). Is there anywhere where I can check that that robots.txt works properly? Is there any chance if this is done correctly of the bots still finding the mirror sites?
Yes the bots can find the site if it is linked to by any other site - but if you disallow googlebot and perhaps the other biggies you will be ok. You can check your robots.txt with Brett's validator here [searchengineworld.com...]