Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Tricky Robots Txt Help,

Two sites one robot

1:12 pm on Dec 20, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 20, 2004
votes: 0

Hi All

I currently have two sites (white label) that share the same files and server directory, however I don't want the second site to be seen by search engine spiders as I know this can be very bad for me with Search Engine listings (duplicate information and mirror sites, etc).

So I thought I might be able to do this with my robots.txt file, it's just how is the problem.

Let's say site one is called (A) www.whatever.co.uk and the other (B)(www.whatevermore.co.uk (fictional sites...hopefully!)

So site B shares the same files as site A so I can't exclude any files or folders as I need these indexing for site A. Site B purely says 'if site = B then show text for site B instead of text for site A.

Can I do this with robots.txtt? Or do you have something better I could implement.

Like I said at the beginning I dont't want site B to be spidered in anyway.

Many thanks and seasons greetings to you all,


1:34 pm on Dec 20, 2004 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
votes: 6

Welcome to WebmasterWorld [webmasterworld.com], badboybart.

This is not something that you can do with robots.txt alone. You have two options: the first is to use mod_rewrite to map robots.txt to a script which will determine the HTTP_HOST name and echo the appropriate robots.txt information. The second way is if you are using server-side scripting within the site, where you detect the HTTP_HOST and add a

<meta name="robots" content="none">
to the page head.
2:50 pm on Dec 20, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 20, 2004
votes: 0

Thanks encyclo (sounds like an X-Men name..?) :O)

I will try the first option as the second, from my understanding and reading through forum, not all spiders follow the tag <meta name="robots" content="none">.

Thanks for your speedy reply (the boss is pleased)