Welcome to WebmasterWorld Guest from 54.235.46.164

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Block test version of site from indexing but keep it available for testing?

     
5:21 pm on Jun 11, 2008 (gmt 0)

Junior Member

10+ Year Member

joined:July 25, 2006
posts:65
votes: 0


Hi, we have a second version of our site called ,
#*$!xTest.com , where the URLs seem to be getting indexed and I am worried it will appear as duplicate content and duplictae pages for the entire site. What do I have to do to block the bots from indexing #*$!xTest.com and the sub Test URLs. If I 404 or redirect it than we can't really use it for testing.

Thank you in advance. J

7:06 pm on June 11, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


Use a robots.txt file in the root folder of your test domain - with one rule:

User-agent: *
Disallow: /

That will keep ALL well-behaved spiders out so you won't get indexed. After a short period, any urls that were indexed should also be dropped.

7:17 pm on June 11, 2008 (gmt 0)

New User

10+ Year Member

joined:Apr 22, 2005
posts: 13
votes: 0


Use .htaccess to set a password.
7:23 pm on June 11, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


Yes, Ferro9 has a good idea - do that, too. It will also keep out the bad bots and scrapers.
11:57 pm on June 11, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


The robots.txt file stops spidering by well behaved robots, but does not disallow the URL for the resource from appearing in the SERPs.

It also does not guard against a scraper stealing all your content before your real site goes live.

I usually disallow in .htaccess/.htpasswd. That stops everyone who does not know the password.

12:33 am on June 12, 2008 (gmt 0)

Junior Member

10+ Year Member

joined:July 25, 2006
posts:65
votes: 0


Thanks