Google: More 301 woes about www and no-www

deleted the 301 and now all pages are back.

[cleanup]

Yes thats right.

After nearly a year of waiting for Google to follow and correct the non-www via a 301 redirect, the only result I had seen was that ALL my pages disappeared from the index.

I waited and waited 10 months is a lot of money down the drain and all my advertisers lost.

So in desparation I deleted the 301 redirect two weeks ago, I figured that whatever happened it could not be any worse that having zero pages in the index.

Result.

The Bad news;
All the pages are back in the index under the non-www listing.

Good News,
All pages are ranking as they were before I tried the redirect and have their original page ranks etc.

So for the moment although far from happy that I can not list my pages in the Google index under www I will settle for having them under the non-www version if I can revive the site and get some traffic back that way.

Anyway just thought I would relay my experience as someone smarter than I may be able to make some sense of it all!

Cheers,
Cleanup

[edited by: cleanup at 7:17 pm (utc) on July 3, 2006]

[K199a]

As for the indexing problems.. why some index as www and some as non-www.. I think it depends on how it was indexed.. And personally I think the Google toolbar is spying on what people type or what websites they go to. People often won't bother with the www's so if you have no links to non-www surely this is the only way they must be indexing such things. Also one of my personal sites got indexed some-how despite having NO links from anywhere and me not submitting it or anything. In fact I've had a job getting it back out.. google eventually did (ish).. yahoo is still indexing half of it.. however i changed the content if it comes from a search engines to something rather blank and robot.txt 'd it to go away. Now i only have one page in google (just the main url.. not sure how to dissallow that one)!?!? and 'just' six in yahoo now. Hmm.. one day it will look at the robot.
Also one thing to note.. google and yahoo are the only places my site has been indexed.. and coincidently are the only toolbars i've installed. Hmmm.

As for the 301.. I think there is a big difference between 301'ing pages within a site and 301'ing pages that Google thinks is a different site. It likes to keep everything in 'different' sites very separate, and unfortunately Google thinks the www and non-www are totally different sites. I am now playing with 301-ing some pages I've moved folders and see how easy that works. Hopefully since Google considers it the same site it should work and I wont have the same problem you guys are unfortunatly having.. but we'll see.

[edited by: K199a at 9:27 am (utc) on July 14, 2006]

[Martin40]

If I knew the site, I might be able to help diagnose more. I guess that's out o' bounds though..

So it seems, unfortunately. I appreciate Googleguy's involvement, but his post suggests that 6/27 is merely due to a conscious effort by Google to achieve something, as expressed in an algorithm.
However, so many white hat sites got hit by this and so many shady sites were unaffected that it just doesn't add up. Since the 6/27 problem one of my pages has been restored in the SERPs, so where's the logic?
If we knew it'd be over in 2-3 weeks then these threads wouldn't even be worth the effort. The current situation has a paralyzing effect - you feel totally unmotivated to work on your site and develop it further. And if it persists, these Google bugs kill sites that took years of people's lives to create. This is not just a error of programming, but also of communication.
I'm sorry if I sound prickly but it's how I feel at the moment.

[edited by: Martin40 at 3:37 pm (utc) on July 14, 2006]

[mcavic]

Come on, Google, I think you guys should be able to figure out 301 redirects. No?

[texasville]

Well, after reading all thru this thread I am now very red-faced.
Some months ago, I did 2 new sites for two businesses. I was tired of the host I had been using and opened a relationship with a new hosting company. A fairly large one. Right afterwards I discovered I would have no access to the .htaccess file. And there was no 301 redirect. This was before I uploaded the sites.
So, I contacted them and explained the reason for the 301 redirect and included the statements from GG to reinforce my claim. To make a long story short...they did a 301 redirect for ALL sites on their hosting services.
Now, I wonder if I did a poor service to everyone connected. And how would I back that out with the host now?
I think I am going to be sick.

[Dayo_UK]

texasville

I know how you feel.

Google said this was the way to deal with this problem and most of the webmaster community has been following this advice - but with what Cleanup and other experiences are showing then perhaps this was not always the way to go.

[mcavic]

I think you shouldn't back it out. Using a 301 to redirect from non-www to www is completely correct. However, any hosting provider should be able redirect or not redirect individual domains based on customer choice.

If search engines aren't interpreting 301's properly, it's the search engines that need to be fixed.

[arubicus]

"I am now playing with 301-ing some pages I've moved folders and see how easy that works."

From our experience it does work and works rather quickly in that the new versions are picked up in the matter of the next crawl. But keep in mind that it may matter how google finds the page. If you put the new url on your site and remove the old then it may crawl that newer version FIRST then it may take alot of time before it decides to refresh the old page and crawl it as a 301. This could lead to a few hiccups and delays. In the case of a whole site redirect (like to different file extensions or even www non-www) it may take forever to get all the old url's refreshed since those urls are likely not linked on site anymore. You would pretty much have to wait until googbot find the old url again and crawl that until it is content.

One thing that may happen is one or the other version goes supplemental and possibly raise some cain until google figures it out -- which may take some time (seemingly forever). The larger the scale of the changes the longer it may take. This can wipe out a site in that the old version (having PR) no longer exists (according to the 301 redirect code) and thus moved to a new location. If it takes too long for google to recognize the relationship between the two and pass PR and other factors then you may see PR tank on the old version being supplemental and having no internal links. On a large scale this can tank a site for a while.

In the case of www non-www if you redirect from non-www to www the same effect can happen. You are saying that the non-www no longer exists because it has moved to the www version and that those two pages are the same thing. If there is no PR on the non-www version you are saying that this no PR page is the same as the www version in that order. So the non-www version goes supplemental and the new version is displayed by those factors of that supplemental version until all factors are passed. So your www version goes to PR 0 and bango your site tanks. Others who have decent PR across both versions would not see such a huge drop. Europeforvisitors site comes into mind on this one.

Over the year the old supplementals have disappeared as far as the www thing goes. Looking pretty clean -- once the old non-www supplementals disappeared things have been moving forward for us. Slowly but still progress. I rarely see gbot hit pages as non-www anymore. This is a good sign as we are finally see some corrections and PR filter throught the site.

We made some file extensions changes on a bunch of pages and much of those are still in there supplemental. We removed the old urls when we made the change (kinda forced to at the time) and replaced them with the new and it has taken forever to get those crawled through. We noticed that google has been crawling some old supplemental pages that are lingering (sup. refresh?) so we thought to help them out by adding the old urls back into our on-site sitemap and Gbot has been working real hard crawling those. Once those get corrected, and I believe they will, I think the BULK of our problems will be solved because of that. The rest will just take time for google to clear up the rest. Since we seen improvements when the old non-www supplementals have been crawled and "removed" I believe this will also have to take place with the old file extensions.

But I believe that this whole thing started out innitially when we did the redirect. It just has taken forever to correct. There are other things we have done, combined with all of the changes google has made, plus a PR drop probably has delayed things further.

Since PR is coming back and google has been crawling like mad on a regular basis. Thing that need to be crawled are being crawled. Just holding onto hope here and looking at this in a more positive light.

TO ADD:
I just remembered that last year when we dropped we seen old urls that redirect to new urls cached and indexed with content of the page the old url redirects to -- yes the 301 was properly in place. This was just another thing I noticed that could have caused problems. I haven't seen this since then but keep an eye out for it.

[Tami]

I have been 301 redirecting non-www to www for my site for at least a year now.

When I perform a "site:www.mysite url.com -www" the majority of my pages come up in google (over 600 pages). All supplemental.

If I do "site:www.mysite url.com www" I get 12 results (including my index page) all not supplemental.

My question is should I be redirecting to the non-www version of my domain or am I reading these results wrong?

Is there another test I should do?

Thanks
Tami

[steveb]

That -www result is not working for anybody in the way it should so don't do anything based on that.

[arubicus]

" That -www result is not working for anybody in the way it should so don't do anything based on that."

Yep steveb is right on this. It is messed up.

[JeffOstroff]

One thing I noticed too after we cleaned up our www vs non www with the .htaccess, I saw 3 of our keywords were at #4, so I sent the links to these Goolge searches to one of our guys to show him, and he said when clicks on them, he sees, rankings of #25, #16, #18, but none of the keywords he saw were ranked #4 like I see. Even when he replied back to me with the links, I click them and I still see #4 rank, and we live in the same city, but I guess we are accessing different DCs.

Seems strange the DCs would be that far off. So how can we tell wich data center our search is going through?

[g1smd]

Use the ShowIP extension on Mozilla/FireFox to see the IP address.

Google has more than 600 active IP addresses, and at least 4 different versions of their index in play spread across them.

[cleanup]

Quick update on my 301 situation.

As you will see from my first post I was having some luck with removing the 301 and getting pages back, albeit under the non-www.

well, about a week ago I noticed that all the pages dissapeared.

Then they started to come back as www.

Great I thought (things are working like they did in the old days before anyone had heard of a 301 redirect )

At first just a few www versions of my pages and then more and more during the week.

Suddenly I noticed a non-www being indexed.

Oh s!"·t!

It looked like the same thing was happening all over again ie the non-www and www being indexed by Google and then subsequent Supplemental hell.

Soo... I rushed around like a mad thing putting that little ol' 301 redirect back in place.

Just in time, I hope.

I managed to catch the process before it was too late and so right now I have just the two non-www pages that Google was able to index and all the rest are www....hooray! about time.

I have seen the pages being indexed bit by bit over the last week.

I now have about 95% of the site indexed correctly.

Things seem to have worked out for the moment.

Observations;
The pages rank more or less where they did before the supplemental fiasco last September.

The www pages have PR0, however the pages seem to rank according to the PR that the non-www pages had.

Questions;
This whole process seems makes me wonder the following;

Did the removal of the 301 in some way kick start this process? (it sure seems like it for me because before the removal there was ten months of stagnation)

Would the site have been indexed to www correctly during this last data push (Data push re MC) just leaving the 301 in place?

Will the site stay www?

Will the site keep its ranking with all pages PR0?

Well anyway as I have said before I think we as webmasters can not hope to answer all the questions as we only have half the story.

Just thought I would report the facts for those interested.
Cleanup

[Panic_Man]

For anyone on a windows server using ASP, here's the non-www to www 301 redirect code I use (usually in a 'config' file that is 'include'd for every page)

if instr(lcase( request.servervariables("SERVER_NAME") ),"www") = 0 then
Response.Status = "301 Moved Permanently"
strNewLocation = "http://www.mydomain.ext" & StripDefaultDoc( request.servervariables("PATH_INFO") )
if len(request.servervariables("QUERY_STRING")) then strNewLocation = strNewLocation & "?" & request.servervariables("QUERY_STRING")
Response.AddHeader "Location", strNewLocation
end if

Function StripDefaultDoc( ByVal sURI )
sURI = Replace( sURI, "/main.asp", "/", 1, 1, vbTextCompare )
sURI = Replace( sURI, "/default.asp", "/", 1, 1, vbTextCompare )
sURI = Replace( sURI, "/default.html", "/", 1, 1, vbTextCompare )
sURI = Replace( sURI, "/default.htm", "/", 1, 1, vbTextCompare )
sURI = Replace( sURI, "/index.asp", "/", 1, 1, vbTextCompare )
sURI = Replace( sURI, "/index.html", "/", 1, 1, vbTextCompare )
sURI = Replace( sURI, "/index.htm", "/", 1, 1, vbTextCompare )
StripDefaultDoc = sURI
End Function

This has always worked for me and usually takes about 6-8 weeks to work completely.

For moving pages to other URLs I use Evolved Code's Smart404 Handler [evolvedcode.net]; from which the above code is derived.

[edited by: Panic_Man at 11:01 pm (utc) on July 15, 2006]

[Stefan]

It needs to be noted that non-www to www might not be the best direction for some websites. If you suspect canonical problems, first look into which version the SE's currently prefer. If it is without a www, consider redirecting the incorrect www subdomain versions to the non-www. Most importantly, don't do anything until you have it all figured out.

[g1smd]

Is the preference for which one has the most number of pages correctly indexed, or for which version the root is doing best as?

Faced with the root index page listed as www, and the non-www as URL-only, supplemental, or missing, but many more pages of the site being listed as non-www compared to the very few listed as www, which way would you point the redirect?

Maybe Matt Cutts will give some guidance on that one?

I don't have any sites in that situation - but I see plenty of people that do...

[Alaskaman]

Man oh man did I step in it by mistake. When I installed Google Analytics for my site Analytics had problems tracking a visit if any link switched from a www. page to a non www. or vice versa. I have a complex custom ecommerce site and hooking up Analytics properly with my site was a pure b.

Along the way in dealing with the www and non-www issue we decided that we had to standardize every link one way or the other if we wanted accurate analytics data. Since we had a little of both we decided to go with no www because most people don't think about www. That might have been correct in terms if common usage, but Google had our home page in their index as www.

We did a proper 301 redirect with mod_rewrite and changed every page and link within the site to non www. Bad move. We should have gone with all www due to how we existed in the Google index.

OK, the damage is done. We did it back in late February. My Google site map shows 35 urls for the current non www urls, and 861 for the www urls. The missing pages are all in Google's supplemental results under the www. urls.

Yahoo has ALL of our site pages indexed without www correctly. No problem for them. MSN has about half of them indexed correctly, which isn't bad for MSN.

I am inclined to say to heck with it since I am a heavy PPC user anyway and organic serps are not as important to me in terms of conversions, although I want them of course. What would be the pros and cons of going backwards to all www. pages at this point with another 301 redirect?

[edited by: Alaskaman at 5:30 am (utc) on July 16, 2006]

[Stefan]

Is the preference for which one has the most number of pages correctly indexed, or for which version the root is doing best as?

This is primarily a guess about the most successfull outcome, but I would base it on the pages that currently bring in the highest most-pertinent traffic. As you suggest, though, some guidance on this from G would be helpful.

[g1smd]

I believe that "which index page (www or non-www) is doing best" is the key to all this, but it would be useful for Google to speak up before too many other people kill their site off by doing the wrong thing.

[Stefan]

Yeah, you might well be right on that g1smd. I really don't know, myself. I took care of the canonical stuff three years ago, after seeing my index page suddenly go missing (had no idea what the hec was going on until I did much reading-up), and missed out on the total mess things became for others since. Sure have read lots of tales of woe in the last couple of years, though.

but it would be useful for Google to speak up before too many other people kill their site off by doing the wrong thing.

Couldn't agree more. Over the last year there have been too many examples of people totally destroying their serps by by trying to fix canonical issues. I think it's best for WW members to frequently remind people newly arrived to the situation that they have to think things through completely before they do anything. And really, since it's G in particular that has such a problem with this, they could offer some suggestions on the best route.

[tr95]

In June 2005, I implemented 301 redirects from non-www to www on all of my websites. In addition, I have been feeding Google daily xml site maps since the beginning of the Google sitemaps program. Yet for one site, Google still has non-www verions of 3 pages (out of about 150) in the index. The www versions of these three pages are not in the Google index. Google crawls some pages of the site every day.

I checked Google's caches of the non-www pages and found dates of January and February 2005. One of the pages has been updated at least 20 times since January 2005, and the others have been updated a few times. I checked Google's caches of about 10 pages that are indexed properly, with www, and found all with dates within the past three weeks.

Thus, for this site at least, the reason that Google is still showing a few non-www pages more than a year after implementation of a 301 redirect is that for those pages, Google is using an ancient cache.

[Dayo_UK]

>>>>I believe that "which index page (www or non-www) is doing best" is the key to all this, but it would be useful for Google to speak up before too many other people kill their site off by doing the wrong thing.

My belief too.

We are starting to see stories now of sites that have followed Googles recommendations on this and have come out the other side with no improvements or in worse shape.

[bull]

site:www.site.net +www

returns perfectly ordered results on a 1300-page content site that suffers since march.
Having had a 301 non-www to www redirect on this site for years, I today started cloaking, presenting Googlebot for non-www pages a 200 OK with robots=noindex and only a link to the www version.
In fact, I already thought of moving my stuff to a new domain - but dismissed that idea. Getting a dozen of DMOZ listings and some hundred links changed, only because the noobs at the plex are unable to handle redirects?
No.

[zeus]

i just added a new site 2 weeks ago and is now listed with non www, it has no non www links to it and all links within are with www.

[Stefan]

i just added a new site 2 weeks ago and is now listed with non www, it has no non www links to it and all links within are with www.

That's bizarre. You're absolutely sure that there were no inbounds without a www? By "all links within are with www", that means you have absolute internal links? You did take care of the canonical stuff with htaccess/whatever when you first put it online, eh?

[g1smd]

That sounds like a site that has an internal 302 redirect, instead of the required 301 redirect from non-www to www.

[edited by: jatar_k at 12:55 pm (utc) on July 17, 2006]

[Stefan]

Thus, for this site at least, the reason that Google is still showing a few non-www pages more than a year after implementation of a 301 redirect is that for those pages, Google is using an ancient cache.

It's 3 pages out of 150 on the site doing this, though. As an experiment, you could try giving those 3 pages some fresh internal links from pages that are crawled daily and well-listed, and see what happens. Pages that are a few clicks away from the index sometimes don't get noticed as often and could have just got lost in the shuffle.

[patiodragon]

Hello all,

I'm not sure I understand exactly what happened in the original post, but I share with you this experience.

When I first got my own domain name, I saw low rankings in the search engines for two different pages on my site: a "www" version and another, "non-www" page.

After some research, creating the following .htaccess file in the top-level web directory has seem to stopped this problem.

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.kimbriggs\.com
RewriteRule ^(.*)$ [kimbriggs.com...] [R=permanent,L]

Hope this helps,
KB

[tr95]

>>>>It's 3 pages out of 150 on the site doing this, though. As an experiment, you could try giving those 3 pages some fresh internal links from pages that are crawled daily and well-listed, and see what happens. Pages that are a few clicks away from the index sometimes don't get noticed as often and could have just got lost in the shuffle.

Two of the three pages are linked directly from the home page, which is crawled almost every day. I suspect that more up-to-date versions with www appear in Google at some times, perhaps most of the time, but today when I checked the old non-www versions happened to come up.

A lot of odd, old stuff is showing up in Google now. Having noticed the large number of 404 error pages in the Google index, I tried searching for my unique domain names and 404 and found a 404 page that is more than a year old, pre-dating my custom 404 page for the site.

[Hissingsid]

Hi,

I have a little site that I've not been able to put a 301 redirect on, its on a Windows server and the host has been unwilling to sort it for me.

All of the links on the site are full absolute URLs.

When I search site:www.mydomain.com I get exactly the same page listing as for site:mydomain.com except the index page URL in the serps omits the www. in the second case.

This site does very well on Ask and quite well on Yahoo it does nothing on Google appearing in the 60s plus range. Since Big Daddy it has occassionally appeared on the first or second page of SERPs on Google for terms where I would say it should be in the top 20. Perhaps this is indicative of some experiment being conducted by Google on live data.

Sid