Forum Moderators: goodroi

Message Too Old, No Replies

blocking the hostsite but not the hosted sites

         

messemaeker

10:05 pm on Sep 5, 2007 (gmt 0)

10+ Year Member



I would like to create a robots.txt-file so that the complete site won't be indexed, but the subdirectories are all the main folders for other sites (other domains).

So;
examplehostdotcom/
shouldn't pop up in google but
examplehostedsitedotcom/
should
but examplehostdotcom/hostedsite/
contains the data for
examplehostedsitedotcom/

so my question is; if I put
User-agent: *
disallow: /
in the robots.txt file, will the other domain be indexed or will the robots check the examplehostdotcom/robots.txt file first?

edit: removed links

goodroi

11:56 am on Sep 7, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Welcome to WebmasterWorld messemaeker!

This is a common problem for people that are hosting multiple domains with only one account. If the pages are being published in both folders than you can block one folder and be ok.

If the pages are being "mirrored" and any change in one folder is repeated in the other then you may not be ok. Since you may end up blocking everything.

My experience is it is best to speak directly with your hosting company and explain what you want. They will know the best how they set up the server. Don't forget to test it using Google's robots.txt validator to ensure everything is doing what it should be.

messemaeker

12:12 pm on Sep 7, 2007 (gmt 0)

10+ Year Member



Thanks. Will do.

UPDATE: this was the response:

Dear Customer, unfortunately it's not possible to configure everything for you not to have your site indexed through Google. robots.txt could really block all
sites from being indexed.So this variant is not possible.

Mmmmmm....