Google showing 301ed URLs in SERPs

[idolw]

I am seeing a weird behaviour for some of our old pages.

When we started the site we did everything manually for a few years. URLs for some important pages were written by hand. Then we set up a CMS which did URLs by certain rules. The old URLs were 301ed to new structure and everything was fine. Pages ranked very well.

The last year was a gradual decrease for the site but we did not have time to investigate too much (other projects) and now, when the damage is severe, we started evaluation.

What I see now, is the old pages lost most of their rankings (went from page1 to page 2-20) for multiple extremely competitive keywords. One reason may be links and not much work done in last 20-some months.
However, the weird part is that:
1. Old URLs rank anywhere between page 1 and page 20
2. Some old URLs are presented in the SERPs in an: Anchor text - SiteName.com format
3. Some old URLs are presented with their normal titles
4. Google Webmaster Tools show links to old URLs
5. Cached versions of old URLs show up-to-date content and current date (+/- 3 weeks)

Now, the questions are:
a/ why does Google link to old 301-ed URLs from SERPs?
b/ is there some technical problem?
c/ should we consider removing redirects and start using old URLs again in the hope for improvement in rankings?

Many thanks in advance

[lucy24]

#2 is the hallmark of a page that is roboted-out or otherwise inaccessible, but not marked "noindex". So g### knows that it exists but nothing else.

(Question for those in the know: If a page's links are all marked "nofollow", will the same thing happen?)

Take a closer look at your redirects and make sure all of those old URLs are going where you want them to go. If you're on speaking terms with the outside linkers, ask them to update their links. Try the old URLs in non-standard forms-- with or without www, with explicit "index.html" if it's a directory, with phony parameters-- and make sure everything gets either a 301 or a 410.

You can also go into gwt and have pages or directories removed from the cache and/or index.

[g1smd]

Make sure the 301 redirect really is a 301 and not a 302.

Request the initial URL both with and without www. and check this always invokes a single step redirect to the final URL. Multiple steps will be a big problem.

[idolw]

Thank you for your input Lucy24 and g1smd.

Header check tools (I am not technical) show 301 pointing to new URLs so it looks like we have single step redirect in place.

None of the page is roboted-out and google shows fresh cache of the pages.
Most of links come from old days to old URLs and we are not on speaking terms with 99% of linkers.

Do you think we should consider getting back to old URLs? I guess the key here would be to figure out a way of doing it without any redirects, etc. Still does it make any sense?

[Robert Charlton]

Header check tools (I am not technical) show 301 pointing to new URLs so it looks like we have single step redirect in place.

Have you checked URLs individually, particularly the old urls that are still showing in serps?

Assuming you're talking about competitive serps, it's somewhere between extremely unlikely and impossible that they would rank on their old urls for competitive searches if a header checker shows that these urls have been 301ed, unless Google has decided to show redirected urls it feels might be in some way suspicious or odd.

2. Some old URLs are presented in the SERPs in an: Anchor text - SiteName.com format

What was the query that returned in the above result. Was it perhaps this....?

[anchor text - sitename.com]

I'm reminded of this discussion, in which redirected alias domain names appeared in the serps....

Domain name replaced in SERPS with alias domain name
http://www.webmasterworld.com/google/4327200.htm [webmasterworld.com]

In the discussion, it was noted that one of the reasons the alias domain may have appeared in the serps was because it was one of the query terms. If the query included the domain name, then your situation might be somewhat analogous.


You also asked...

Do you think we should consider getting back to old URLs? I guess the key here would be to figure out a way of doing it without any redirects, etc. Still does it make any sense?

IMO, and this is a total guess... it would make sense only if the redirects were the problem.

If the inbound links to the redirected urls were the problem, then you wouldn't really be changing what's causing the trouble. I'm thinking after all this time, it doesn't make sense to go back to the old urls.

I'd check quality of inbound links and also really have someone knowledgeable check over those redirects.

[idolw]

Thank you for your time, Robert Charlton.

Have you checked URLs individually, particularly the old urls that are still showing in serps?

Assuming you're talking about competitive serps, it's somewhere between extremely unlikely and impossible that they would rank on their old urls for competitive searches if a header checker shows that these urls have been 301ed, unless Google has decided to show redirected urls it feels might be in some way suspicious or odd.

I am talking about widget reservation site for major destinations. Is that competitive enough?
Yes, I checked the URLs separately, Google shows www.website.com/oldurl with _Anchor text - website.com_ title, and when I click, I am instantly redirected to www.website.com/newurl.

What was the query that returned in the above result. Was it perhaps this....?

No, the queries I use are those that I try to rank for, i.e. [destination widgets], [widgets in destination] in my case.

IMO, and this is a total guess... it would make sense only if the redirects were the problem.

If the inbound links to the redirected urls were the problem, then you wouldn't really be changing what's causing the trouble. I'm thinking after all this time, it doesn't make sense to go back to the old urls.

I'd check quality of inbound links and also really have someone knowledgeable check over those redirects.

I will have an eye of someone properly technical tomorrow and they'll look into it.
The links to the old URLs were not of high quality (most of them come from 2004-5 link swap mania) and we did not really build any links in the last 3-5 years. New URLs may have acquired just a handful of links each as the vertical is highly commercial (i.e. few people link out).

[idolw]

Hello Everyone,

A small update in my case.
We checked the redirects with an IT guys and here's what we receive when we used Fiddler debugger:

HTTP/1.1 301 Moved Permanently
Cache-Control: private
Content-Length: 0
Location: http://www.domain.com/NewUrl
Server: Microsoft-IIS/7.5
X-AspNet-Version: 2.0.50727
Machine: ServerName
Set-Cookie: www.domainname.com&expiresDate=2013-07-26 16:08:17; domain=.com; expires=Fri, 26-Jul-2013 14:08:17 GMT; path=/
X-Powered-By: ASP.NET
Date: Thu, 26 Jul 2012 14:08:17 GMT

There definitely is only one-step redirect for each and every URL.

What I noticed however, is
- when I make a [site:domain.com inurl:newurl] search in Google it does not show anything.
- when I make a [site:domain.com inurl:oldurl] search in Google it does show the requested page but in Anchor text - Domain Name format.

Any more ideas? Maybe there is something wrong with our CMS? But what?
.

[edited by: Robert_Charlton at 6:56 pm (utc) on Jul 26, 2012]
[edit reason] disabled auto-linking and smilies to make post legible [/edit]

[g1smd]

Check the HTTP responses for both www and non-www for each of those for both the old and the new URL.

[idolw]

Here are HTTP responses for all situations:
OldURL www
HTTP/1.1 301 Moved Permanently
Content-Length: 272
Content-Type: text/html
Location:

http://www.domain/NewURL

Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Fri, 27 Jul 2012 12:22:43 GMT
Connection: close

OldURL non-www
HTTP/1.1 301 Moved Permanently
Content-Length: 272
Content-Type: text/html
Location:

http://www.domain/NewURL

Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Fri, 27 Jul 2012 12:24:45 GMT
Connection: close

NewURL www
HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 58104
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 2.0.50727
Machine: ServerName
Set-Cookie: U=; domain=.domain.com; expires=Thu, 26-Jul-2012 12:25:53 GMT; path=/
Set-Cookie: www.domain.com=lang=en&expiresDate=2013-07-27 14:25:53; domain=.domain.com; expires=Sat, 27-Jul-2013 12:25:53 GMT; path=/
X-Powered-By: ASP.NET
Date: Fri, 27 Jul 2012 12:25:52 GMT

NewURL non-www
HTTP/1.1 301 Moved Permanently
Cache-Control: private
Content-Length: 0
Location:

http://www.domain/NewURL

Server: Microsoft-IIS/7.5
X-AspNet-Version: 2.0.50727
Machine: ServerName
X-Powered-By: ASP.NET
Date: Fri, 27 Jul 2012 12:26:20 GMT

[Robert Charlton]

Hmmm.... Try checking your source code to see if you're using any of the old urls in your nav links.

[idolw]

Thank you, Robert.
All URLs are generated by the CMS so all links in the code are new ones (checked manually, too).

We found an issue with Title length descrbed here: [seroundtable.com...]
Googler John Mu suggests to decrease the URL length to 66 characters as this is the number of words Google displays in the SERPs. Our URLs are around 80 characters. Funny thing, however, is that other pages of the same site that were created later and have always had the New URLs show in the SERPs with proper titles.
We will decrease titles of the problematic pages to see if it helps.

[phranque]

have you tried to "fetch as googlebot" (in GWT) the old urls that are indexed?

perhaps googlebot is getting a 302 instead of a 301.

[idolw]

hi Phranque, yes, we did. We get the same responses as posted above.

[Robert Charlton]

Trying to guess what might be causing Google to return the wrong url in the serps, even with a 301 in place... and it occurs that the canonical link element might be involved.

Do you by any chance have canonical link elements on your pages? If you do, double-check them to make sure they're correct.