homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

how to manage cname in robot
don't was to be indexed

10+ Year Member

Msg#: 3506398 posted 5:31 am on Nov 16, 2007 (gmt 0)


I have a site where i have used 2 of the cname like cname.domain.com and cname2.domain.com. Now i cna see that these cnames urls are indexed and i dont want them in engines at all. how should i stop these to be indexed through robot file. is something other way except this.

help appreciated in advance..





10+ Year Member

Msg#: 3506398 posted 9:17 am on Dec 14, 2007 (gmt 0)

cud anybody tell me pls...

i have cname made for my domain and i dont want them in engines.

should i block the cname like this

disallow [cname.domain.com...]

Will this work or this will also effect the parent site domain.com..?

thanks a lot for sugestions.



10+ Year Member

Msg#: 3506398 posted 7:58 am on Dec 21, 2007 (gmt 0)


ThomasB & goodroi,




WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 3506398 posted 2:10 pm on Dec 21, 2007 (gmt 0)


Please be aware that this is a forum, not instant messaging. Due to differences in time zones, you may be posting when the person best able to answer your questions is sleeping. :)

Also, the moderators are not primarily responsible for replying to posts, they are responsible for moderating posts.

Unless the CNAMES map to a different document root -- in other words, unless they each have their own file space on your server and serve different pages than your primary domain, then robots.txt won't help you. This is because robots.txt assumes that each domain/hostname has it's own robots.txt file, and it has no provisions for specifying which domain/hostname the restrictions that it contains apply to.

If your CNAMES do map to separate filespace, then you can put a unique robots.txt into the root directory for each CNAME, containing
User-agent: *
Disallow: /
to disallow all robots from fetching all resources.

However, in many cases, all hostnames are 'pointed' to the same file space, and a unique robots.txt for each domain can't be used. The usual solution is to redirect all non-canonical domain name variants to the main domain using a 301-Moved Permanently redirect. This can be done with mod_rewrite or PHP on Apache, ISAPI Rewrite or .asp on IIS, and several other methods, depending on your server technology. A search for "redirect canonical domain" on WebmasterWorld may turn up some useful threads to help get you started.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved