I have a question for which the answer may be blindingly obvious to the more experienced here. ..hopefully..
I really would appreciate help here, as I am quite concerned about duplicate content being indexed by SE spiders.
I have just transferred our domains to a new site. The directories are set up so that there is the first publicly viewed level where I keep no files but a few directories with various programs that are administrative, and four directories which host each of our four domains on diff unique IP's. via the directory pointing method.
Normally I would put a robots.txt in each of the domain directories. So far so good.
However the top level directory is also accessable by a URL such as [www27.ispprovidername.com...] And that URL if extended would also point to our actual registered domains - [www27.ispprovidername.com...] [www27.ispprovidername.com...] to dir4/.
Now I want the search engine spiders to spider all of our domain directories (the second level down), but I dont want them to spider the top level directory and the non-domain pointed admin directories. My concern is that they will index duplicate content once from [www27.ispprovidername.com...] to dir 4 and once from [ourdomain.com...] resulting in 2 listings of the exact same content.
However if I put a robots.txt in there disallowing access to that directory itself and the admin directories will it probibit spiders from visiting all the 4 domain pointed directories in deeper directories - e.g [www27.ispprovidername.com...] [www27.ispprovidername.com...] etc
Sorry if this is garbled but ive tried to make it as clear as I can.