1. Home
  2. Forums Index
  3. Deprecated / (deprecated) Google News Archive 6:57 am Apr 24, 2026

Forum Moderators: open

Message Too Old, No Replies

Disallowing robots on www2 and www3 servers

Can that affect the main www site also?

         

sdani

2:02 am on Mar 17, 2004 (gmt 0)

10+ Year Member



I have three different servers at three different hosts. The main site is www.mysite.com and the others are www2.mysite.com and www3.mysite.com.
I had same index page on all three and my main site was made "Supplimental" because of duplicate content. I changed the index pages on www2 & www3 to redirect to www page with 301 for about a month and nothing changed. Then I modified the index pages on www2 & www3 3 weeks ago and still no luck with google. The www page is still being shown as supplimental result.
During past 5-6 weeks, I was able to get some very good PR5-6 links (around 20) from different edu sites, where people actually visited the site and gave me links based on the merits of the site.
As of today's PR update -> the index is still PR0 and is still listed as "Supplimental Result"

My Question:
Apart from the index page, I only host dynamic scripts on my www2 & www3 servers. All the content is hosted only on www server. So, to avoid the duplication, if I DISALLOW * on my www2 & www3 servers, would that affect crawling of my www server by Google?
I checked Google's robots.txt on www & www2 servers and they are exactly the same? Not sure why other search engines don't bar them for duplicate pages.

kaled

2:58 pm on Mar 17, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Alternatively, place the following in the <HEAD> of the index pages on www2 and www3.

<META NAME="robots" CONTENT="noindex">

or possibly

<META HTTP-EQUIV="refresh" CONTENT="0;URL=http://www.yourdomain.com/">

This sort of redirect is supposedly understood by Google. Haven't tried it myself. I don't think it's appropriate in your case, but it's your call.

Kaled.

sdani

8:28 pm on Mar 17, 2004 (gmt 0)

10+ Year Member



Thanks Kaled:

This is an excellent suggestion to just put noindex on other two index pages. But:
1) I have "NoIndex" option on my privacy policy page, but Google still lists that page in site:mysite.com, without title and description. So may be by the time the crawler would know not to index this page, it has already crawled it.
2) You mentioned "Alternatively", so do you think my proposal of adding Disallow * on www2 & www3 servers would also work without causing any harm to www site?

Thanks
SDani

kaled

10:01 pm on Mar 17, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The global disallow should be ok but it's all guesswork where Google's algos are concerned.

Thinking about it more, perhaps the redirect I mentioned might be appropriate.

I'm certainly not qualified to give definitive advice. With anything you try, you'll still need to cross your fingers and hope.

Kaled.

sdani

10:05 pm on Mar 17, 2004 (gmt 0)

10+ Year Member



Thanks, I will try that.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

  1. Home
  2. Forums Index
  3. Deprecated / (deprecated) Google News Archive 6:57 am Apr 24, 2026