Welcome to WebmasterWorld Guest from

Forum Moderators: mademetop

Message Too Old, No Replies

robots.txt file to prevent indexing and avoid site penalties

or how to split a site and point it at 2 geo-zones when content is similar



4:46 pm on Sep 28, 2009 (gmt 0)

5+ Year Member

Hi everyone,

I'm developing a new site using some of the same products and text thatís already on an existing .co.uk site. I then plan to point one at the UK and the other at the USA - the .com version of the same url. Whilst certain products are the same, page coding, key text, spelling, is changed in the two versions.

As I understand Google doesn't like duplication of sites with the same material, and possibly punishes you for it, I understand I can use a robots.txt file to prevent Google from indexing the new site in the interim.

Can someone tell me what I should put in this file and where I should upload it to? Do I need to create a directory called Ďrobotsí? Can I place this text file on some pages only (those with duplication) and let others be indexed that have new material?

As a related issue, but of equal importance, I obviously wish to have Google recognize the existence of the site (and have placed it on my Google dashboard accordingly), in order that the 6-month incubation period ticks away. So I donít want to put the clock back to Day One.

Thanks in advance for your much-valued advice and help.



1:06 pm on Oct 8, 2009 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

the robots.txt file goes in the root directory.

this google webmaster tools help page will have some useful information to get you started:
Block or remove pages using a robots.txt file - Webmasters/Site owners Help [google.com]


Featured Threads

Hot Threads This Week

Hot Threads This Month