homepage Welcome to WebmasterWorld Guest from 54.237.134.62
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

This 41 message thread spans 2 pages: 41 ( [1] 2 > >     
Weird subdomains in Google SERP, how do I get rid of these?
meowattack




msg:4635620
 10:42 pm on Jan 6, 2014 (gmt 0)

I have a whole bunch of weird subdomain like vww.mysite.com pop.mysite.com www.mysite.com showing up in Google when I search site:http://mysite.com. About 14 of them are being indexed by Google and I am hoping to remove them because we don't use them and I'm worried they are hurting our website's overall SERP ranking in Google. What is the safest way to get rid of these? I asked our host goDaddy but they haven't helped and when I look at the DNS manager options in goDaddy, not all of these weird subdomains are even shown in there as aliases.

Thanks for any advice!

 

lucy24




msg:4635649
 1:03 am on Jan 7, 2014 (gmt 0)

Do you use any subdomains at all? If not, you can let your existing domain-name canonicalization redirect take care of it temporarily:

RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

The long-term fix does involve DNS settings, which someone else will explain. But if you're having trouble, you can start redirecting instantly. Where I said "www.example.com", substitute the exact preferred form of your domain name.

bumpski




msg:4635754
 3:07 pm on Jan 7, 2014 (gmt 0)

Using the rel="canonical" meta tag on all your pages should clear this up in a month or two, also.
<link rel="canonical" href="http://mysite.com/..pathtopages.html">

Make sure your rel="canonical" insertion code (if you use it), handles https (SSL) requests as well. Unbeknownst to many webmasters, many web hosts allow direct access to a website's full content through "https".

meowattack




msg:4635866
 2:04 am on Jan 8, 2014 (gmt 0)

Thanks for the help, but I think I must need to do something in DNS settings because we do have some subdomains that are used, just not any of the ones that are showing up in Google.

my_name




msg:4635877
 7:26 am on Jan 8, 2014 (gmt 0)

Not sure if it's related but pages on one of my web sites is being indexed under a domain of another web site on the same VPS.

For example...

Page should be: domain.com/segment/segment
Page indexed: mail.domain2.com/segment/segment

The segments are correct but are being accessed by sub domain on another domain. I've never had this happen before.

Please let me know if you find a fix! :)

lucy24




msg:4635881
 8:11 am on Jan 8, 2014 (gmt 0)

pages on one of my web sites is being indexed under a domain of another web site on the same VPS

Yikes. If you can communicate with your host, do so. It could be something nefarious-- or it could be a technical problem.

How did you find out that you were being indexed under someone else's name?

phranque




msg:4635885
 8:59 am on Jan 8, 2014 (gmt 0)

welcome to WebmasterWorld, meowattack!


you need to add a hostname canonicalization redirect to your configuration.

something like this (assuming apache and .htaccess):
RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

my_name




msg:4635915
 1:00 pm on Jan 8, 2014 (gmt 0)

@lucy24 I have contacted my host but they said it must be a DNS issue and couldn't advise on what to do (not very helpful). I've thought about using my own nameservers but seems like a lot of hassle for something that has never happened before. I own the other domain that it is listed with in Google... I searched for a headline on the website and that's when I discovered it :S

lucy24




msg:4636009
 7:36 pm on Jan 8, 2014 (gmt 0)

Whew. I thought you'd discovered a scraper living on your very own server. If you control the other domain, then phranque's fix should work nicely.

Are your registrar and DNS the same people?

Robert Charlton




msg:4636035
 9:32 pm on Jan 8, 2014 (gmt 0)

Getting back to meowattack's issue...

meowattack - phranque's post on hostname canonicalization describes the proper solution for your problem, but my guess is that you don't have access to .htaccess on your GoDaddy account.

I'm thinking that you might have access to your DNS A-Name settings... and that they're currently set up permitting what's called "wildcard DNS". It's often indicated on your DNS control panel with an asterisk, as in *.example.com.

While this makes setting up subdomains much easier, it can expose you to a lot of mischief... because essentially it enables any subdomain to be returned.

How many other subdomains do you have, and how are they set up?

lucy24




msg:4636039
 10:13 pm on Jan 8, 2014 (gmt 0)

my guess is that you don't have access to .htaccess

Is this really true? If so, you're looking at a different fix: change hosts :)

phranque




msg:4636056
 11:31 pm on Jan 8, 2014 (gmt 0)

it's not necessarily a wildcard DNS issue.
pop.mysite.com and mail.domain2.com would be typical hostnames used for POP3 and SMTP servers, so they may need to exist in the DNS configuration if you are using email on those domains.
the problem occurs when the IP address of the mail servers also responds to HTTP requests.
the web server is where the wildcard subdomain configuration certainly exists.
you can either disable wildcard subdomains and let those HTTP requests timeout without a connection or you can accept those requests and respond with a redirect to the canonical hostname.

meowattack




msg:4636082
 3:48 am on Jan 9, 2014 (gmt 0)

I can edit .htaccess with my shared goDaddy linux web server.

These are the subdomains showing up in Google:

mail.
smtp.
w.
ww.
www.www.
pop.
mx1.
vww.
imap.
pop.www.
mail.www.
mx1.www.
imap.www.

These are the CNAME (Alias) records in goDaddy:

e points to email.secureserver.net
email points to email.secureserver.net
ftp points to @
imappoints to imap.secureserver.net
mailpoints to pop.secureserver.net
mobilemail points to mobilemail-v01.prod.mesa1.secureserver.net
pdapoints to mobilemail-v01.prod.mesa1.secureserver.net
poppoints to pop.secureserver.net
smtp points to smtp.secureserver.net
webmailpoints to webmail.secureserver.net
wwwpoints to @

And then under a different section in the goDaddy DNS manager is MX Mail Exchanger:

10@mailstore1.secureserver.net
0@smtp.secureserver.net

Which I think explains the mx1. subdomain.

We do use subdomains files. and time. which are not represented in goDaddy and are controlled somewhere else, I've asked my colleague to send me the list of legitimate sudomains in addition to those. That is why I'm worried that phranque's suggestion
RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

will interfere with those other needed subdomains?

What I don't understand is what is causing the w. ww. wwv. and the doubling up www.www. The other ones seem to be caused by the goDaddy record.

phranque




msg:4636095
 5:30 am on Jan 9, 2014 (gmt 0)

what does the A record look like?

meowattack




msg:4636230
 8:10 pm on Jan 9, 2014 (gmt 0)

In goDaddy our A record is:

@ points to IP Address
testsite points to same IP address.

But then I got ahold of our account where our domain is registered and there we have our desired subdomains and also these A records:

www
@ (None)
* (All Others)

Is that * (all others) a wildcard setup that I should remove? I also read that I could use my htaccess file to create 301 redirects for all of these subdomains to the main www site. Here: [moz.com...]

lucy24




msg:4636259
 10:22 pm on Jan 9, 2014 (gmt 0)

I also read that I could use my htaccess file to create 301 redirects for all of these subdomains to the main www site.

See phranque's post about halfway up this thread.

It is probably better not to allow wildcard subdomains unless you genuinely need them-- for example if you let users create their own subdomains in the form myname.example.com/

meowattack




msg:4636284
 12:59 am on Jan 10, 2014 (gmt 0)

I see lucy24 (#1) and phranque (#2) gave these .htaccess modifications but I don't think they are appropriate in my situation because I can't redirect all subdomains. We have subdomains that we actually use. See #3 for what I'm trying to find.

#1
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

#2
RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

#3
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.example\.com$ [NC]
RewriteCond %{HTTP_HOST} ^mail\.example\.com [NC]
RewriteCond %{HTTP_HOST} ^smtp\.example\.com [NC]
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]
</IfModule>

I need to be able to direct the www version of my site to the non-www version and then I thought there might be a way to add a rewrite condition for each subdomain that shouldnt exist to 301 redirect to also the non-www version of my site. The above that I tried doesn't work, still looking.

[edited by: phranque at 3:26 am (utc) on Jan 10, 2014]
[edit reason] exemplified domain [/edit]

JD_Toims




msg:4636290
 1:54 am on Jan 10, 2014 (gmt 0)

You don't need to allow HTTP requests to mail. or smtp., unless you serve content that needs to be available via HTTP request from them -- If there is, then I suggest moving it to the main version of the domain rather than on the subdomains -- .htaccess only affects HTTP requests, not mail, ftp or anything else, so you should be fine redirecting everything to a single version of your domain in your .htaccess file.



There's actually a typo in the code Lucy24 and phranque posted as examples.
The opening on the right side of the condition should be a ( not a {.

[mod's note]typo edited in all places above to avoid future cut-and-paste recidivism[/mod's note]


RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

meowattack




msg:4636303
 3:13 am on Jan 10, 2014 (gmt 0)

Ok I tried that to see if it messed with any of my subdomains but it didnt do the redirect. Could these other parts of my.htaccess file being interfering?
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_HOST} !^(mysite\.com)?$
RewriteRule (.*) http://mysite.com/$1 [R=301,L]
</IfModule>

# BEGIN W3TC Page Cache core
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP:Accept-Encoding} gzip
RewriteRule .* - [E=W3TC_ENC:_gzip]
RewriteCond %{REQUEST_METHOD} !=POST
RewriteCond %{QUERY_STRING} =""
RewriteCond %{REQUEST_URI} \/$
RewriteCond %{HTTP_COOKIE} !(comment_author|wp-postpass|w3tc_logged_out|wordpress_logged_in|wptouch_switch_toggle) [NC]
RewriteCond %{HTTP_USER_AGENT} !(W3\ Total\ Cache/0\.9\.3) [NC]
RewriteCond "%{DOCUMENT_ROOT}/wordpress/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index.html%{ENV:W3TC_ENC}" -f
RewriteRule .* "/wordpress/wp-content/cache/page_enhanced/%{HTTP_HOST}/%{REQUEST_URI}/_index.html%{ENV:W3TC_ENC}" [L]
</IfModule>
# END W3TC Page Cache core
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

.

[edited by: Robert_Charlton at 3:27 am (utc) on Jan 10, 2014]
[edit reason] delinked mysite.com - use example.com, which doesn't autolink [/edit]

phranque




msg:4636304
 3:18 am on Jan 10, 2014 (gmt 0)

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.example\.com$ [NC]
RewriteCond %{HTTP_HOST} ^mail\.example\.com [NC]
RewriteCond %{HTTP_HOST} ^smtp\.example\.com [NC]
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]
</IfModule>

- the IfModule container is unnecessary.

- literal periods in regular expressions should be escaped with a backslash.

- consecutive RewriteCond conditions are ANDed by default and you want to OR those conditions.
however with all those wildcard subdomains already indexed it will make the conditionals list unwieldy and growing.

- it would be better if you weed out all the files.example.com and time.example.com content for redirects before canonicalizing your main domain.

how would you recognize requests that should be redirected to the files and time subdomains?

meowattack




msg:4636313
 4:22 am on Jan 10, 2014 (gmt 0)

The fact that I have multiple sites like projects.example.com and time.example.com is why redirecting all subdomains seemed incorrect, since I believe those would be HTTP requests. Thats why I wanted to do the conditionals list with the specific ones I wanted gone. Are you saying doing the conditional list is a bad idea?

I think I'm unclear as to what you mean by "it would be better if you weed out all the files.example.com and time.example.com content for redirects before canonicalizing your main domain."

lucy24




msg:4636315
 4:32 am on Jan 10, 2014 (gmt 0)

How many subdomains have you got? Ignore the ones like mail. and smtp. Only look at the ones that are meant to be reachable by http.

We do use subdomains files. and time. which are not represented in goDaddy and are controlled somewhere else, I've asked my colleague to send me the list of legitimate sudomains in addition to those.

What do you mean by "represented" and "controlled"? If you mean that they physically live on a different server, then you do not need to do anything about them in your primary htaccess. Now, if there's a DNS hiccup and people arrive at your server looking for a subdomain that lives somewhere else, that's a separate redirect.

Use example.com to prevent auto-linking. Unfortunately this is no help in subdomains. So locutions like
http://subdomain.example.com
will need some additional jiggery-pokery.

meowattack




msg:4636605
 10:58 pm on Jan 10, 2014 (gmt 0)

##### IP CONTROLLED WITH .htaccess ######
www
@ (None)
* (All Others)
3d.example.net
testsite.example.net
www2.example.com

IP #2
files.example.net
ftp.example.net
files.example.com
ftp.example.com

IP #3
licenses.example.net
licenses.example.com

IP #4
projects.example.net
projects.example.com
xilinx.example.com

IP #5
time.example.net
time.example.com

So based on this and what you guys have said, 301 redirecting all subdomains in the web server's .htaccess file shouldn't have any effect on the other subdomains on other ip's.

I put the following at the top of my .htaccess file:
RewriteEngine On
RewriteCond %{HTTP_HOST} !^(example\.com)?$
RewriteRule (.*) http://example.com/$1 [R=301,L]

But I'm not getting redirected when I try any of these that are being indexed by Google:
mail.
smtp.
w.
ww.
www.www.
pop.
mx1.
vww.
imap.
pop.www.
mail.www.
mx1.www.
imap.www.

JD_Toims




msg:4636630
 1:42 am on Jan 11, 2014 (gmt 0)

But I'm not getting redirected when I try any of these that are being indexed by Google:

Have you tried with an empty cache or a "clean" browser -- One of the "things" that comes along with browsers all trying to be the fastest is they're reluctant to let go of caches in some cases, so make sure when testing anything .htaccess related it's with an empty cache or "unused/clean" browser.



Also, an idea is to try to "fetch as Googlebot" from within WMT and see what Google's getting to make sure it's not cache related -- Google "caches", but doesn't "cache for requests" -- Google "keeps a copy" and then compares it to "a new request" to see if anything's changed and/or how much since the last request for a resource.

lucy24




msg:4636639
 2:03 am on Jan 11, 2014 (gmt 0)

301 redirecting all subdomains in the web server's .htaccess file shouldn't have any effect on the other subdomains on other ip's

Exactly. An htaccess file is concerned with physical filepaths, not hostnames. If a request is pointed to some other server in the first place-- or even a different userspace on the same server-- it will never "see" the htaccess.

But I'm not getting redirected when I try any of these that are being indexed by Google:

That's actually good, because it means the requests aren't going to the server. But they must be going somewhere, or else the request would simply time out and you'd get a browser error message. (Something like "The server isn't responding and I have stopped trying.") Where are these requests being handled?

Can you pinpoint a timeframe for the bad indexing? Maybe a one-day server hiccup that pointed everyone to the wrong place?

meowattack




msg:4637482
 10:37 pm on Jan 14, 2014 (gmt 0)

@JD_Toims, Yep I've tried on different browsers after emptying the cache. I haven't tried the fetch as Googlebot, but I believe that would require me to setup each subdomain in WMT and prove to google I manage those subdomains. I'll look into this.

@lucy24: I'm not sure what you mean by where the requests are being handled. I believe they are being handled by my server. When I put in any random subdomain, for example hithere.example.com it returns the same page everytime, a little graphic that says "Sorry this site is not currently available". I'm assuming its getting this .jpg from my server (im about to go look for this) and that someone in the past setup the server to respond this way.

Also I'm getting a 200 rather than a 404 when I use this HTTP request and a <server header checker>
.

[edited by: Robert_Charlton at 12:09 am (utc) on Jan 15, 2014]
[edit reason] removed specific tool link, per forum Charter [/edit]

lucy24




msg:4637509
 1:46 am on Jan 15, 2014 (gmt 0)

When I put in any random subdomain, for example hithere.example.com it returns the same page everytime, a little graphic that says "Sorry this site is not currently available". I'm assuming its getting this .jpg from my server (im about to go look for this) and that someone in the past setup the server to respond this way.

What I'd really like to know is whether random subdomains from random other domains (ones that have nothing to do with you) on the same server lead to the same graphic. That is: Is it your site's own jpg, or is it something the host pops in when there's a wildcard request?

Something, somewhere is instructing the server to put up this jpg.

Is it a page that contains a jpg (.html extension in browser's address bar, even if the jpg is its only content), or just the jpg by itself (jpg extension)?

meowattack




msg:4637643
 4:45 pm on Jan 15, 2014 (gmt 0)

Its an html. It appears that maybe goDaddy is providing this page. In the title tags there is default.secureserver.net.

When I googled this, I found other sites with the same page and jpg image.

I'm still confused as to why, even if goDaddy is providing this, my .htaccess doesn't override this behavior.

JD_Toims




msg:4637653
 4:55 pm on Jan 15, 2014 (gmt 0)

default.secureserver.net

Because your .htaccess isn't read for the domain above.

meowattack




msg:4637713
 10:17 pm on Jan 15, 2014 (gmt 0)

So if when I try to go to a subdomain like mail.example.com, my DNS had wildcards allowed, so it sends the browser to my site at goDaddy. But then is goDaddy intervening before the browser can arrive at my server and redirecting to this default.secureserver.net page?

This 41 message thread spans 2 pages: 41 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved