Forum Moderators: Robert Charlton & goodroi
I, like many others here, have been a victim of the Google re-indexing that took place late last year (a number #1 site is now on the 2nd or 3rd results pages, with many irrelevant or ghost sites before me).
After months and months of investigating and pulling my hair out, I am really no closer to figuring out why Google suddenly does not like this particular site anymore.
One of the things I came across today is the way Google indexes two versions of a site URL.
For example, if I type in the URL of the above mentioned site as:
www.site.com
It will return with a fully indexed title, description, cache date etc, of the page.
However, if I type in just
site.com
It returns with just the following (hyperlinked):
site.com
with no description, title or cache date.
Now, is this normal? Not always it seems. I used another site I own and did the same experiement, and in both cases Google returned with a full description, cache date etc for both versions of the url.
Now this got me thinking - have we done something from a system admin point of view to the site I mentioned first above, that may effect the way Google looks at and indexes one version of the url vs the other?
Bottomline - what reasons would Google have to not fully index site.com Vs www.site.com
Would love some opionins from experienced webmasters here.
A site check and backlink check show the same results - I just seem to have no description etc for the non-www version in Google. - and the www version is now showing a relatively old cache (16th Feb)
How many times do we have to hear that before someone knows what the heck is going on. What kind of company doesn’t let its subscribers know if there is a problem? You know they can not satisfied with these results. Please talk about this very specific cluster of site factors. What is causing this? Are your sites new?
Personally I hope MSN is the future.
Bottomline - what reasons would Google have to not fully index site.com Vs www.site.com
Why is it that I only buy one domain domain.com and not two domains?
I don't have two copies of my content pages out there, one for www.domain.com and one for domain.com.
It seems to me to be another instance of silly search engine issues that the average site owner (me) shouldn't have to worry about.
I did ask my hosting company about redirecting everything to www and they said they couldn't do it (windows server).
There is great danger in saying that www.site and site are the same. Remember the incredible problems caused by them assuming site and site/index.html, site/index.asp, and site/index.php were all the same - that let people get their competitors delisted for 404 errors so easily?
<added>
This appears to be another instance of links to you can have a detrimental affect on your site.
My content is the same (I guess like most websites)
But would this be enough for Google to drop my rankings. Surely most websites are set up like this. I dont have any other pages indexed without the www - just the homepage which shows no title and description and is a pr0.
I have now redirected the non-www to the www (about a month ago and have seen no improvements yet - as mentioned my cache is 16th Feb)
Anyone had experience of this in the past and examples of how long it has taken for improvements after the redirect from www to non-www - I have seen 1 or 2 months quoted a couple of times.
and message 9 of the following thread:-
[webmasterworld.com...]
I wonder if it is time to rebuild (?) this data again.
I am simply pointing out that is some cases Google's listings for them are different, but in some cases Googgle's listings for them are exactly the same.
My question is not whether this is morally right or wrong and whether I think I'm greedy for two listings or not. My question is directed at why some sites are listed twice for both url versions (both site.com and www.site.com fully listed), and some are listed differently (site.com fully listed, but www.site.com not) - and if this potentially has something to do with the site's ranking value.
Thanks Claus, I actually requested the 301 when I originally read that thread and I gave them that exact info... perhaps they don't know enough about how to set it -
If you run a server header check on your site it will say what kind of server it's on and version # then maybe you can get help in the Apache thread for that server. If it's something like Ils then there is a workaround for it in that thread.
I certainly know nothing. But i still think it isn't a problem that I should have to deal with since I have no control over who links to me and how they link to me. All my internal links are fully expanded and consistent.
Only if that were true. However now we have to either take care of it ourselves or hope the world starts using another search engine.
Your first step should be to immediately set up htaccess to redirect non-www requests to www with a 301.
Your second step should be to use absolute URLs instead of relative URLs for your links.
If someone had linked to the site as [example.com,...] but all the internal links had been to www pages, the damage would have been limited.
--
I put on a 301 redirect from the non www to the www version nearly 3 months ago but google still lists both.
You might want to check the server headers to make sure that your 301 is set up correctly - there is a tool in your account's control panel (bottom left) for checking this.
--
what kind of company doesn’t let its subscribers know if there is a problem?
How does one "subscribe" to Google? It makes no sense to blame Google - as it could be just as likely that MSN or Yahoo would crawl the non-WWW as well - blaming them for your oversight makes no sense.
Yes I found 2 sites which had linked to our non www version and got them to change it around Xmas time. In January I put a 301 redirect from the non www version to www (the header checkers I have used does pick up the 301 when i search for non www) and now use absolute urls and slowly google is getting rid of the non www pages (at one stage I had around 300 non www ones but now its down to 40) but it still lists my homepage for www and non-www (both without title and desc). I have 1000's of links pointing to the www one - would a link to the non www one help so google finds the 301 or is it better to wait (been nearly 3 months since the 301 redirect has been there)- my site is in my profile if anyone is able to look and give their opinion.
I have the same problem on more than one site.
I have put the 301 in place and have set up a link to the non-www version on a site to see what happens - Googlebot has not crawled it yet though (:() - Will let you know what happens.
I think (suspect) google finds the non-www even without a link from anywhere sooner or later.
I certainly have domains that I have never linked to and never used appearing with no title and description in the index.
I think we may have exchanged links once :)
I have got a site with no title and description on both non-www and www - i think that non-www might take over as the primary domain (cant remember Googleguys words canonical url - thats it) and as the non-www is so low on backlinks it does not get crawled as frequently and therefore becomes no title and description on both. :(
Hope that the re-direct sorts us both out.
November was when I was hit too.
Dec update things got worse, February even worse and still not recovering.... was yours a similar decline in traffic?
did you want to sticky me your URL?
RewriteEngine on
RewriteCond %{HTTP_HOST}!^www\.yourdomain\.com [NC]
RewriteRule ^(.*)$ http://www.yourdomain.com/$1 [R=301,L]
This will send any requests to [yourdomain.com...] and forward them to [yourdomain.com...]
Additionally, we made sure that all internal links to folders always ended in a trailing / so that the server does not have to redirect from folder to folder/. For links to index pages, never include the filename, end at the folder name with a trailing / on the URL.
After a few days, many of the non-www pages started to lose their title and snippet. To help this along we did a Google site: search and pulled out all of the non-www URLs. We then set up a page on another site that linked to all of those.
Within a few days the number of www pages indexed had gone from 150 to over 200 and nearly all of them have a title and description. Additionally, the number of non-www pages has risen from 60 to 110 but ALL of them do NOT have a title or snippet (the number has probably risen simply because many of those internal pages have never been linked to from outside, and Google is now indexing deeper than it did before, but each time it finds a new deep page it also sees that it serves a 301 too - with the result that it adds the URL but does not add the title or snippet for it).
I expect all of the non-www pages to drop out of the index very shortly.
The Contractor - have done - but the domain without the www takes ages to get crawled and therefore as Wizard suggested me and stickyboy are wondering if linking to the domain without the www will get Googlebot to pick up and resolve the problem sooner.
At this rate I might have to get some more links to non-www domain.
I did the Google search with &num=100&filter=0 on the end of the Google search URL to get 100 results at a time. I saved each page to disk. I then used a text editor to join all the saved pages together to make one huge HTML file. To extract the URLs that I needed, I ran Xenu LinkSleuth over the file and let it generate a report. I then loaded that HTML report into my editor and chopped out just the part that said "List of valid URLs you can submit to search engines". I saved that as a new HTML page and uploaded it to the server.
Yes, I have linked to the page I want the 301 to be picked up but Googlebot just does not want to crawl it.
I have been checking the logs with a fine toothcombe for Googlebot to pick up the 301.
Are you seeing Googlebot in the logs going to domain.com/whatever.html picking up the 301 and then going to www.domain.com/whatever.html - that is what I am looking for (I think.) Slurp is doing it (not that I have a problem with Yahoo at the moment.)
Perhaps the page I am linking from is not strong enough. I hate linking my own sites together and avoid it like the plague.
[edited by: Dayo_UK at 3:57 pm (utc) on Mar. 25, 2005]
The Contractor - have done - but the domain without the www takes ages to get crawled and therefore as Wizard suggested me and stickyboy are wondering if linking to the domain without the www will get Googlebot to pick up and resolve the problem sooner.
Well, if you are waiting for Google to discard pages in the index you have a long wait. I had taken down a site and Google kept the pages in the index and kept ranking the site for over a year....hehe
Added: as a matter of fact I just did a search and my old the site I took down over a year ago still comes up in 2nd place on 2 word term with 2,960,000 results and it's only a contact form for selling the domain.... such is the power of links...
My understanding is not that the page will not be discarded but that it will become a 301 redirect and therefore become the same page in Googles eyes.
eg if I do a link:domain.com and a link:www.domain.com I get the same results. and if I do a site:domain.com and a site:www.domain.com I get the same results etc.
Am I barking up the wrong tree?
I would not use the Google URL removal especially with the modrewrite as you may remove the www pages :)
