homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Block test version of site from indexing but keep it available for testing?

 5:21 pm on Jun 11, 2008 (gmt 0)

Hi, we have a second version of our site called ,
#*$!xTest.com , where the URLs seem to be getting indexed and I am worried it will appear as duplicate content and duplictae pages for the entire site. What do I have to do to block the bots from indexing #*$!xTest.com and the sub Test URLs. If I 404 or redirect it than we can't really use it for testing.

Thank you in advance. J



 7:06 pm on Jun 11, 2008 (gmt 0)

Use a robots.txt file in the root folder of your test domain - with one rule:

User-agent: *
Disallow: /

That will keep ALL well-behaved spiders out so you won't get indexed. After a short period, any urls that were indexed should also be dropped.


 7:17 pm on Jun 11, 2008 (gmt 0)

Use .htaccess to set a password.


 7:23 pm on Jun 11, 2008 (gmt 0)

Yes, Ferro9 has a good idea - do that, too. It will also keep out the bad bots and scrapers.


 11:57 pm on Jun 11, 2008 (gmt 0)

The robots.txt file stops spidering by well behaved robots, but does not disallow the URL for the resource from appearing in the SERPs.

It also does not guard against a scraper stealing all your content before your real site goes live.

I usually disallow in .htaccess/.htpasswd. That stops everyone who does not know the password.


 12:33 am on Jun 12, 2008 (gmt 0)


Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved