homepage Welcome to WebmasterWorld Guest from 54.166.255.168
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Block test version of site from indexing but keep it available for testing?
jaimes

5+ Year Member



 
Msg#: 3672292 posted 5:21 pm on Jun 11, 2008 (gmt 0)

Hi, we have a second version of our site called ,
#*$!xTest.com , where the URLs seem to be getting indexed and I am worried it will appear as duplicate content and duplictae pages for the entire site. What do I have to do to block the bots from indexing #*$!xTest.com and the sub Test URLs. If I 404 or redirect it than we can't really use it for testing.

Thank you in advance. J

 

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3672292 posted 7:06 pm on Jun 11, 2008 (gmt 0)

Use a robots.txt file in the root folder of your test domain - with one rule:

User-agent: *
Disallow: /

That will keep ALL well-behaved spiders out so you won't get indexed. After a short period, any urls that were indexed should also be dropped.

Ferro9

5+ Year Member



 
Msg#: 3672292 posted 7:17 pm on Jun 11, 2008 (gmt 0)

Use .htaccess to set a password.

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3672292 posted 7:23 pm on Jun 11, 2008 (gmt 0)

Yes, Ferro9 has a good idea - do that, too. It will also keep out the bad bots and scrapers.

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3672292 posted 11:57 pm on Jun 11, 2008 (gmt 0)

The robots.txt file stops spidering by well behaved robots, but does not disallow the URL for the resource from appearing in the SERPs.

It also does not guard against a scraper stealing all your content before your real site goes live.

I usually disallow in .htaccess/.htpasswd. That stops everyone who does not know the password.

jaimes

5+ Year Member



 
Msg#: 3672292 posted 12:33 am on Jun 12, 2008 (gmt 0)

Thanks

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved