Forum Moderators: open

Message Too Old, No Replies

Subdomain indexed with robots.txt disallowing?!

         

zelv

2:33 pm on Dec 3, 2010 (gmt 0)

10+ Year Member



I hide my aff links that I use in mysite.com at go.mysite.com and together with index.php, where I redirect to advertiser, I have a robots.txt like that:

User-agent: *
Disallow: /

Thing is: google indexed my subdomain, how is this possible?

Thanks in advance

dstiles

10:00 pm on Dec 3, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Is the sub-domain in the same root folder on the server?

Sub-domains are completely different from the parent domain unless redirected to them. www.mysite.com is a sub-domain but usually redirected to the primary domain mysite.com.

If both primary domain and sub-domain(s) are using the same root folder (ie robots.txt is in both site roots) then you have cause for complaint.

zelv

10:42 pm on Dec 3, 2010 (gmt 0)

10+ Year Member



I treat my sub-domain as a completely different domain.

I have my main domain in a folder at public_html, like "mysite" folder, and I have my sub-domain exactly on the same place: in another folder at public_html, like "go.mysite" folder. I have no redirection at all, sub-domain is intended to be a different domain for what I need.

I have a robots.txt inside the folder "go.mysite.com" which is disallowing any spider, like

User-agent: *
Disallow: /

And my sub-domain is indexed by Google!

dstiles

6:52 pm on Dec 5, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Do you have a WMT account with google? If so, what does it say about robots?

I assume your sub-domain addresses the home page (eg) go.mysite.com/index.html - if it's go.mysite.com/folder/index.html it's wrong and google will index it.

"no redirection at all"... I assume you have one record on the web server set up for mysite.com pointing to the primary web site and another record pointing to the sub-site for the sub-domain; eg

mysite.com - public_html
go.mysite.com - public_html_go

If you type go.mysite.com/robots.txt what do you get? You should see the file contents in your browser.