homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Restrict Specific Domain Names from Being Indexed
Using the robots.txt file on Google to restrict certain domains...

 12:23 am on Oct 29, 2003 (gmt 0)

Hi there all :)

I have a website at www.olddomain.com that was listed on Google. I have recently been able to purchase a better domain name that has great keywords - www.newdomain.com.

I would like Google to spider the site, but only list the www.newdomain.com domain as I do not want to get penalised for spamming multiple addresses. Basically we use www.olddomain.com for print and www.newdomain.com for web promotion.

Any ideas?

Thanks in advance.

- Ben Slater

[edited by: heini at 9:43 am (utc) on Oct. 29, 2003]
[edit reason] Please don't post real urls or domain names / Thanks! [/edit]



 12:45 am on Oct 29, 2003 (gmt 0)

If you're on Apache, you can use mod_rewrite to redirect robots.txt requests to different files based on the requested domain name using {HTTP_HOST}. On other servers, you could use a script to do essentially the same thing.

So, one domain appears to have a robots.txt that allows spidering, and the other one disallows spidering.

See [webmasterworld.com...]



 1:36 am on Oct 29, 2003 (gmt 0)

Hmmmm, thanks for the advice Jim. I will follow up with my webhost to see if they can help...it's good to go in with a little bit of knowledge so I don't look like a complete lamer ;)

- Ben

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved