Block test version of site from indexing but keep it available for testing?

[jaimes]

Hi, we have a second version of our site called ,
#*$!xTest.com , where the URLs seem to be getting indexed and I am worried it will appear as duplicate content and duplictae pages for the entire site. What do I have to do to block the bots from indexing #*$!xTest.com and the sub Test URLs. If I 404 or redirect it than we can't really use it for testing.

Thank you in advance. J

[tedster]

Use a robots.txt file in the root folder of your test domain - with one rule:

User-agent: *
Disallow: /

That will keep ALL well-behaved spiders out so you won't get indexed. After a short period, any urls that were indexed should also be dropped.

[Ferro9]

Use .htaccess to set a password.

[tedster]

Yes, Ferro9 has a good idea - do that, too. It will also keep out the bad bots and scrapers.

[g1smd]

The robots.txt file stops spidering by well behaved robots, but does not disallow the URL for the resource from appearing in the SERPs.

It also does not guard against a scraper stealing all your content before your real site goes live.

I usually disallow in .htaccess/.htpasswd. That stops everyone who does not know the password.

[jaimes]

Thanks