Weird subdomains in Google SERP, how do I get rid of these?

[meowattack]

I have a whole bunch of weird subdomain like vww.mysite.com pop.mysite.com www.mysite.com showing up in Google when I search site:http://mysite.com. About 14 of them are being indexed by Google and I am hoping to remove them because we don't use them and I'm worried they are hurting our website's overall SERP ranking in Google. What is the safest way to get rid of these? I asked our host goDaddy but they haven't helped and when I look at the DNS manager options in goDaddy, not all of these weird subdomains are even shown in there as aliases.

Thanks for any advice!

[lucy24]

Do you use any subdomains at all? If not, you can let your existing domain-name canonicalization redirect take care of it temporarily:

RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

The long-term fix does involve DNS settings, which someone else will explain. But if you're having trouble, you can start redirecting instantly. Where I said "www.example.com", substitute the exact preferred form of your domain name.

[bumpski]

Using the rel="canonical" meta tag on all your pages should clear this up in a month or two, also.

<link rel="canonical" href="http://mysite.com/..pathtopages.html">

Make sure your rel="canonical" insertion code (if you use it), handles https (SSL) requests as well. Unbeknownst to many webmasters, many web hosts allow direct access to a website's full content through "https".

[meowattack]

Thanks for the help, but I think I must need to do something in DNS settings because we do have some subdomains that are used, just not any of the ones that are showing up in Google.

[my_name]

Not sure if it's related but pages on one of my web sites is being indexed under a domain of another web site on the same VPS.

For example...

Page should be: domain.com/segment/segment
Page indexed: mail.domain2.com/segment/segment

The segments are correct but are being accessed by sub domain on another domain. I've never had this happen before.

Please let me know if you find a fix! :)

[lucy24]

pages on one of my web sites is being indexed under a domain of another web site on the same VPS

Yikes. If you can communicate with your host, do so. It could be something nefarious-- or it could be a technical problem.

How did you find out that you were being indexed under someone else's name?

[phranque]

welcome to WebmasterWorld, meowattack!


you need to add a hostname canonicalization redirect to your configuration.

something like this (assuming apache and .htaccess):

RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

[my_name]

@lucy24 I have contacted my host but they said it must be a DNS issue and couldn't advise on what to do (not very helpful). I've thought about using my own nameservers but seems like a lot of hassle for something that has never happened before. I own the other domain that it is listed with in Google... I searched for a headline on the website and that's when I discovered it :S

[lucy24]

Whew. I thought you'd discovered a scraper living on your very own server. If you control the other domain, then phranque's fix should work nicely.

Are your registrar and DNS the same people?

[Robert Charlton]

Getting back to meowattack's issue...

meowattack - phranque's post on hostname canonicalization describes the proper solution for your problem, but my guess is that you don't have access to .htaccess on your GoDaddy account.

I'm thinking that you might have access to your DNS A-Name settings... and that they're currently set up permitting what's called "wildcard DNS". It's often indicated on your DNS control panel with an asterisk, as in *.example.com.

While this makes setting up subdomains much easier, it can expose you to a lot of mischief... because essentially it enables any subdomain to be returned.

How many other subdomains do you have, and how are they set up?

[lucy24]

my guess is that you don't have access to .htaccess

Is this really true? If so, you're looking at a different fix: change hosts :)

[phranque]

it's not necessarily a wildcard DNS issue.
pop.mysite.com and mail.domain2.com would be typical hostnames used for POP3 and SMTP servers, so they may need to exist in the DNS configuration if you are using email on those domains.
the problem occurs when the IP address of the mail servers also responds to HTTP requests.
the web server is where the wildcard subdomain configuration certainly exists.
you can either disable wildcard subdomains and let those HTTP requests timeout without a connection or you can accept those requests and respond with a redirect to the canonical hostname.

[meowattack]

I can edit .htaccess with my shared goDaddy linux web server.

These are the subdomains showing up in Google:

mail.
smtp.
w.
ww.
www.www.
pop.
mx1.
vww.
imap.
pop.www.
mail.www.
mx1.www.
imap.www.

These are the CNAME (Alias) records in goDaddy:

e points to email.secureserver.net
email points to email.secureserver.net
ftp points to @
imappoints to imap.secureserver.net
mailpoints to pop.secureserver.net
mobilemail points to mobilemail-v01.prod.mesa1.secureserver.net
pdapoints to mobilemail-v01.prod.mesa1.secureserver.net
poppoints to pop.secureserver.net
smtp points to smtp.secureserver.net
webmailpoints to webmail.secureserver.net
wwwpoints to @

And then under a different section in the goDaddy DNS manager is MX Mail Exchanger:

[e-mail]
[e-mail]

Which I think explains the mx1. subdomain.

We do use subdomains files. and time. which are not represented in goDaddy and are controlled somewhere else, I've asked my colleague to send me the list of legitimate sudomains in addition to those. That is why I'm worried that phranque's suggestion
RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

will interfere with those other needed subdomains?

What I don't understand is what is causing the w. ww. wwv. and the doubling up www.www. The other ones seem to be caused by the goDaddy record.

[phranque]

what does the A record look like?

[meowattack]

In goDaddy our A record is:

@ points to IP Address
testsite points to same IP address.

But then I got ahold of our account where our domain is registered and there we have our desired subdomains and also these A records:

www
@ (None)
* (All Others)

Is that * (all others) a wildcard setup that I should remove? I also read that I could use my htaccess file to create 301 redirects for all of these subdomains to the main www site. Here: [moz.com...]

[lucy24]

I also read that I could use my htaccess file to create 301 redirects for all of these subdomains to the main www site.

See phranque's post about halfway up this thread.

It is probably better not to allow wildcard subdomains unless you genuinely need them-- for example if you let users create their own subdomains in the form myname.example.com/

[meowattack]

I see lucy24 (#1) and phranque (#2) gave these .htaccess modifications but I don't think they are appropriate in my situation because I can't redirect all subdomains. We have subdomains that we actually use. See #3 for what I'm trying to find.

#1
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

#2
RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

#3
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.example\.com$ [NC]
RewriteCond %{HTTP_HOST} ^mail\.example\.com [NC]
RewriteCond %{HTTP_HOST} ^smtp\.example\.com [NC]
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]
</IfModule>

I need to be able to direct the www version of my site to the non-www version and then I thought there might be a way to add a rewrite condition for each subdomain that shouldnt exist to 301 redirect to also the non-www version of my site. The above that I tried doesn't work, still looking.

[edited by: phranque at 3:26 am (utc) on Jan 10, 2014]
[edit reason] exemplified domain [/edit]

[JD_Toims]

You don't need to allow HTTP requests to mail. or smtp., unless you serve content that needs to be available via HTTP request from them -- If there is, then I suggest moving it to the main version of the domain rather than on the subdomains -- .htaccess only affects HTTP requests, not mail, ftp or anything else, so you should be fine redirecting everything to a single version of your domain in your .htaccess file.

---

There's actually a typo in the code Lucy24 and phranque posted as examples.
The opening on the right side of the condition should be a ( not a {.

[mod's note]typo edited in all places above to avoid future cut-and-paste recidivism[/mod's note]


RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

[meowattack]

Ok I tried that to see if it messed with any of my subdomains but it didnt do the redirect. Could these other parts of my.htaccess file being interfering?
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_HOST} !^(mysite\.com)?$
RewriteRule (.*) http://mysite.com/$1 [R=301,L]
</IfModule>

# BEGIN W3TC Page Cache core
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP:Accept-Encoding} gzip
RewriteRule .* - [E=W3TC_ENC:_gzip]
RewriteCond %{REQUEST_METHOD} !=POST
RewriteCond %{QUERY_STRING} =""
RewriteCond %{REQUEST_URI} \/$
RewriteCond %{HTTP_COOKIE} !(comment_author|wp-postpass|w3tc_logged_out|wordpress_logged_in|wptouch_switch_toggle) [NC]
RewriteCond %{HTTP_USER_AGENT} !(W3\ Total\ Cache/0\.9\.3) [NC]
RewriteCond "%{DOCUMENT_ROOT}/wordpress/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index.html%{ENV:W3TC_ENC}" -f
RewriteRule .* "/wordpress/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index.html%{ENV:W3TC_ENC}" [L]
</IfModule>
# END W3TC Page Cache core
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

.

[edited by: Robert_Charlton at 3:27 am (utc) on Jan 10, 2014]
[edit reason] delinked mysite.com - use example.com, which doesn't autolink [/edit]

[phranque]

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.example\.com$ [NC]
RewriteCond %{HTTP_HOST} ^mail\.example\.com [NC]
RewriteCond %{HTTP_HOST} ^smtp\.example\.com [NC]
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]
</IfModule>

- the IfModule container is unnecessary.

- literal periods in regular expressions should be escaped with a backslash.

- consecutive RewriteCond conditions are ANDed by default and you want to OR those conditions.
however with all those wildcard subdomains already indexed it will make the conditionals list unwieldy and growing.

- it would be better if you weed out all the files.example.com and time.example.com content for redirects before canonicalizing your main domain.

how would you recognize requests that should be redirected to the files and time subdomains?

[meowattack]

The fact that I have multiple sites like projects.example.com and time.example.com is why redirecting all subdomains seemed incorrect, since I believe those would be HTTP requests. Thats why I wanted to do the conditionals list with the specific ones I wanted gone. Are you saying doing the conditional list is a bad idea?

I think I'm unclear as to what you mean by "it would be better if you weed out all the files.example.com and time.example.com content for redirects before canonicalizing your main domain."

[lucy24]

How many subdomains have you got? Ignore the ones like mail. and smtp. Only look at the ones that are meant to be reachable by http.

We do use subdomains files. and time. which are not represented in goDaddy and are controlled somewhere else, I've asked my colleague to send me the list of legitimate sudomains in addition to those.

What do you mean by "represented" and "controlled"? If you mean that they physically live on a different server, then you do not need to do anything about them in your primary htaccess. Now, if there's a DNS hiccup and people arrive at your server looking for a subdomain that lives somewhere else, that's a separate redirect.

Use example.com to prevent auto-linking. Unfortunately this is no help in subdomains. So locutions like

http://subdomain.example.com

will need some additional jiggery-pokery.

[meowattack]

##### IP CONTROLLED WITH .htaccess ######
www
@ (None)
* (All Others)
3d.example.net
testsite.example.net
www2.example.com

IP #2
files.example.net
ftp.example.net
files.example.com
ftp.example.com

IP #3
licenses.example.net
licenses.example.com

IP #4
projects.example.net
projects.example.com
xilinx.example.com

IP #5
time.example.net
time.example.com

So based on this and what you guys have said, 301 redirecting all subdomains in the web server's .htaccess file shouldn't have any effect on the other subdomains on other ip's.

I put the following at the top of my .htaccess file:
RewriteEngine On
RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

But I'm not getting redirected when I try any of these that are being indexed by Google:
mail.
smtp.
w.
ww.
www.www.
pop.
mx1.
vww.
imap.
pop.www.
mail.www.
mx1.www.
imap.www.

[JD_Toims]

But I'm not getting redirected when I try any of these that are being indexed by Google:

Have you tried with an empty cache or a "clean" browser -- One of the "things" that comes along with browsers all trying to be the fastest is they're reluctant to let go of caches in some cases, so make sure when testing anything .htaccess related it's with an empty cache or "unused/clean" browser.

---

Also, an idea is to try to "fetch as Googlebot" from within WMT and see what Google's getting to make sure it's not cache related -- Google "caches", but doesn't "cache for requests" -- Google "keeps a copy" and then compares it to "a new request" to see if anything's changed and/or how much since the last request for a resource.

[lucy24]

301 redirecting all subdomains in the web server's .htaccess file shouldn't have any effect on the other subdomains on other ip's

Exactly. An htaccess file is concerned with physical filepaths, not hostnames. If a request is pointed to some other server in the first place-- or even a different userspace on the same server-- it will never "see" the htaccess.

But I'm not getting redirected when I try any of these that are being indexed by Google:

That's actually good, because it means the requests aren't going to the server. But they must be going somewhere, or else the request would simply time out and you'd get a browser error message. (Something like "The server isn't responding and I have stopped trying.") Where are these requests being handled?

Can you pinpoint a timeframe for the bad indexing? Maybe a one-day server hiccup that pointed everyone to the wrong place?

[meowattack]

@JD_Toims, Yep I've tried on different browsers after emptying the cache. I haven't tried the fetch as Googlebot, but I believe that would require me to setup each subdomain in WMT and prove to google I manage those subdomains. I'll look into this.

@lucy24: I'm not sure what you mean by where the requests are being handled. I believe they are being handled by my server. When I put in any random subdomain, for example hithere.example.com it returns the same page everytime, a little graphic that says "Sorry this site is not currently available". I'm assuming its getting this .jpg from my server (im about to go look for this) and that someone in the past setup the server to respond this way.

Also I'm getting a 200 rather than a 404 when I use this HTTP request and a <server header checker>
.

[edited by: Robert_Charlton at 12:09 am (utc) on Jan 15, 2014]
[edit reason] removed specific tool link, per forum Charter [/edit]

[lucy24]

When I put in any random subdomain, for example hithere.example.com it returns the same page everytime, a little graphic that says "Sorry this site is not currently available". I'm assuming its getting this .jpg from my server (im about to go look for this) and that someone in the past setup the server to respond this way.

What I'd really like to know is whether random subdomains from random other domains (ones that have nothing to do with you) on the same server lead to the same graphic. That is: Is it your site's own jpg, or is it something the host pops in when there's a wildcard request?

Something, somewhere is instructing the server to put up this jpg.

Is it a page that contains a jpg (.html extension in browser's address bar, even if the jpg is its only content), or just the jpg by itself (jpg extension)?

[meowattack]

Its an html. It appears that maybe goDaddy is providing this page. In the title tags there is default.secureserver.net.

When I googled this, I found other sites with the same page and jpg image.

I'm still confused as to why, even if goDaddy is providing this, my .htaccess doesn't override this behavior.

[JD_Toims]

default.secureserver.net

Because your .htaccess isn't read for the domain above.

[meowattack]

So if when I try to go to a subdomain like mail.example.com, my DNS had wildcards allowed, so it sends the browser to my site at goDaddy. But then is goDaddy intervening before the browser can arrive at my server and redirecting to this default.secureserver.net page?