homepage Welcome to WebmasterWorld Guest from 23.21.9.44
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
how to manage cname in robot
don't was to be indexed
experienced




msg:3506400
 5:31 am on Nov 16, 2007 (gmt 0)

Hi,

I have a site where i have used 2 of the cname like cname.domain.com and cname2.domain.com. Now i cna see that these cnames urls are indexed and i dont want them in engines at all. how should i stop these to be indexed through robot file. is something other way except this.

help appreciated in advance..

thanks

Exp...

 

experienced




msg:3527516
 9:17 am on Dec 14, 2007 (gmt 0)

cud anybody tell me pls...

i have cname made for my domain and i dont want them in engines.

should i block the cname like this

disallow [cname.domain.com...]

Will this work or this will also effect the parent site domain.com..?

thanks a lot for sugestions.

Exp.

experienced




msg:3532600
 7:58 am on Dec 21, 2007 (gmt 0)

Dear

ThomasB & goodroi,

CUD YOU PLS CARE TO REPLY..

THANKS

jdMorgan




msg:3532756
 2:10 pm on Dec 21, 2007 (gmt 0)

experienced,

Please be aware that this is a forum, not instant messaging. Due to differences in time zones, you may be posting when the person best able to answer your questions is sleeping. :)

Also, the moderators are not primarily responsible for replying to posts, they are responsible for moderating posts.

Unless the CNAMES map to a different document root -- in other words, unless they each have their own file space on your server and serve different pages than your primary domain, then robots.txt won't help you. This is because robots.txt assumes that each domain/hostname has it's own robots.txt file, and it has no provisions for specifying which domain/hostname the restrictions that it contains apply to.

If your CNAMES do map to separate filespace, then you can put a unique robots.txt into the root directory for each CNAME, containing
User-agent: *
Disallow: /
to disallow all robots from fetching all resources.

However, in many cases, all hostnames are 'pointed' to the same file space, and a unique robots.txt for each domain can't be used. The usual solution is to redirect all non-canonical domain name variants to the main domain using a 301-Moved Permanently redirect. This can be done with mod_rewrite or PHP on Apache, ISAPI Rewrite or .asp on IIS, and several other methods, depending on your server technology. A search for "redirect canonical domain" on WebmasterWorld may turn up some useful threads to help get you started.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved