homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
Forum Library, Charter, Moderators: mademetop

General Search Engine Marketing Issues Forum

robots.txt file to prevent indexing and avoid site penalties
or how to split a site and point it at 2 geo-zones when content is similar

5+ Year Member

Msg#: 3997140 posted 4:46 pm on Sep 28, 2009 (gmt 0)

Hi everyone,

I'm developing a new site using some of the same products and text thatís already on an existing .co.uk site. I then plan to point one at the UK and the other at the USA - the .com version of the same url. Whilst certain products are the same, page coding, key text, spelling, is changed in the two versions.

As I understand Google doesn't like duplication of sites with the same material, and possibly punishes you for it, I understand I can use a robots.txt file to prevent Google from indexing the new site in the interim.

Can someone tell me what I should put in this file and where I should upload it to? Do I need to create a directory called Ďrobotsí? Can I place this text file on some pages only (those with duplication) and let others be indexed that have new material?

As a related issue, but of equal importance, I obviously wish to have Google recognize the existence of the site (and have placed it on my Google dashboard accordingly), in order that the 6-month incubation period ticks away. So I donít want to put the clock back to Day One.

Thanks in advance for your much-valued advice and help.




WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 3997140 posted 1:06 pm on Oct 8, 2009 (gmt 0)

the robots.txt file goes in the root directory.

this google webmaster tools help page will have some useful information to get you started:
Block or remove pages using a robots.txt file - Webmasters/Site owners Help [google.com]

Global Options:
 top home search open messages active posts  

Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved