homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Tricky Robots Txt Help,
Two sites one robot

 1:12 pm on Dec 20, 2004 (gmt 0)

Hi All

I currently have two sites (white label) that share the same files and server directory, however I don't want the second site to be seen by search engine spiders as I know this can be very bad for me with Search Engine listings (duplicate information and mirror sites, etc).

So I thought I might be able to do this with my robots.txt file, it's just how is the problem.

Let's say site one is called (A) www.whatever.co.uk and the other (B)(www.whatevermore.co.uk (fictional sites...hopefully!)

So site B shares the same files as site A so I can't exclude any files or folders as I need these indexing for site A. Site B purely says 'if site = B then show text for site B instead of text for site A.

Can I do this with robots.txtt? Or do you have something better I could implement.

Like I said at the beginning I dont't want site B to be spidered in anyway.

Many thanks and seasons greetings to you all,




 1:34 pm on Dec 20, 2004 (gmt 0)

Welcome to WebmasterWorld [webmasterworld.com], badboybart.

This is not something that you can do with robots.txt alone. You have two options: the first is to use mod_rewrite to map robots.txt to a script which will determine the HTTP_HOST name and echo the appropriate robots.txt information. The second way is if you are using server-side scripting within the site, where you detect the HTTP_HOST and add a <meta name="robots" content="none"> to the page head.


 2:50 pm on Dec 20, 2004 (gmt 0)

Thanks encyclo (sounds like an X-Men name..?) :O)

I will try the first option as the second, from my understanding and reading through forum, not all spiders follow the tag <meta name="robots" content="none">.

Thanks for your speedy reply (the boss is pleased)


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved