Google will not follow 301?

Is setting an htaccess 301 redirect a fatal mistake?

[Kukenan]

Our website was nicely indexed in Google and crawled regularly.
However it was indexed only under http://site.com.

Having read some threads in this forum regarding htaccess redirects and how Google would follow them properly I did prepare the move from http://site.com to http://www.site.com.

Now Goglebot is just scratching the surface of my site and every hit I get from it is a 301. except from robots.txt.

This is an average visit example:

64.68.82.54 - - [02/Jul/2003:23:24:54 -0400] "GET /robots.txt HTTP/1.0" 301 240 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

64.68.82.54 - - [02/Jul/2003:23:25:00 -0400] "GET /robots.txt HTTP/1.0" 200 146 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

64.68.82.54 - - [02/Jul/2003:23:25:07 -0400] "GET /anypage-at-anydepth.html HTTP/1.0" 301 242 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

and then it is gone.

My question is:
What can I expect from the next update?
- Will the site show the same data under www.

- or will it just be gone altogether?

Would I be better off submiting the site again under www.site.com

This has been going on for about two weeks now and Google doesn't get where the site is.

Thanks in advance for your comments.

[hakre]

Hi Kukenan,

More info about the 301 Issue and google can be found in the following thread. Some of the people there supported, that 301 is not working as intended.

301 not being supported? ¦ old, deleted pages being indexed? (forum3/6486.htm) [webmasterworld.com]

hope this will help you a bit.
-hakre

[hetzeld]

Hi Kukenan,

I did it a couple of months ago for my main site and Googlebot handled it perfectly. It simply kept both URLs in the index for approximately 6 weeks.
That worried me a bit, because of the temporary duplicate content.

Simply make sure that you have no remaining links without the www in your site and everything should be OK in a few weeks from now. I don't think resubmitting your site would change anything.

Dan

[keyplyr]

My experience is the same as hetzeld.

[DavidT]

I did the same thing more or less, just pointing www.domain.com to domain.com, and my experience was exact same as Kukenan, Googlebot must have made thousands of requests that ended in unfollowed 301s. This went on for at least a month, probably more, but it has sorted itself out fine now, the bot rarely makes a request now with 'www' in the url.

Just takes time I guess.

[Kukenan]

Thank you guys!

It is reassuring to know I have not made the biggest mistake (at least not alone ;-))

It's been nearly a month now and Googlebot has almost visited the entire website.

I figure the process works like this:

1.- The bot has to visit each and every page in the index looking for it in the last known location, it will get a 301 redirect and NOTES the new location (but doesn't clrawl it).

2.- Once it has figured out the new location of all those pages (and I mean all of them) Googlebot is ready to crawl the new pages. It will begin hiting pages asking for the www.version of them. No more 301's.

3.- After it crawls all the pages in the new location the index gets updated properly.

I guess it is just a matter of waiting...

I will post the conclusion to this story. (if any)

[hetzeld]

Hi Kukenan,

For GoogleBot to know that a page has been moved to another location/domain, it has to ask for the old page at least once. So, if your site contains a few thousand pages, you'll have at least the same number of 301 hits.

Dan

[Kukenan]

OK.

Our site is not that big, just barely over 100 pages.

What I would like to see is Googlebot visiting one of the pages in the new location.

So far it has not touched any of them.

After I see it happen at least once, I'll start breathing again....

[kanama]

my experience is this:

we changed from a .com to .co.uk with a 301 reflecting this, this was to get listed on google.co.uk aswell as the .com site since we were hosted in america.

our site wasnt that great in google rankings but when it followed the 301, it did crawl the redirected pages.... but it also took it off the serps too

ok this was to be expected eventually but i would have thought the old comes off and new on straight away. now the redirected pages are off google and the index hasnt been updated leaving us off google for the recently crawled new site too.

[Kukenan]

Exactly Kanama, and that is my worst fear.

I know eventually the site will be back on the index but i am afraid it will stay out of Google's database for two crawling cycles (whatever that means with the "new" Google).

I didn't want to ask for him but .. is Googleguy arround?

[nolen1]

Hello, I'm having the same problem as Kukenan with two of my sites. I have been doing good in Google until yesterday. My sites were indexed as [domain.com...] now they are listed as [domain.com...] . Now I have really dropped in the serps. When I check the number of incoming links I see that the http:// and [www....] are counted seperately. I only have 4 links to [www....] and a lot more than that to [domain.com....]

I have read about peoples experiences with a .htaccess file and I would like to make one. I have never done it before. I have tried several of the examples that I have found here in this forum in notepad and used cuteftp to upload to my site this is what my hosting company told me to do. I must not be doing some thing right because it's not working. Here's an example of what I added:

RewriteCond %{HTTP_HOST}!^domain\.com
RewriteRule ^.*$ [domain.com%{REQUEST_URI}...] [R=301,L]

I would like to direct to [domain.com...] since I have more links and that's how they have always been indexed.
I tried that and others and with www. and it still didn't work. Am I using the right thing and am I supposed to save the file as text and save it in my web? I would greatly appreciate any help. Thank You.

[hetzeld]

Hi Nolen1,

Try this one, as it works for me:

RewriteCond %{HTTP_HOST} !^domain.com
RewriteRule (.*) [domain.com...] [R=301,QSA,L]

Dan

[GoogleGuy]

301 is the right way to go, Kukenan. It takes a little time, as hetzeld and keyplayr mention, but when it goes into effect it should be exactly what you want. Go for it.

[anallawalla]

This definitely works for me:

Options +FollowSymLinks 
RewriteEngine on 
RewriteCond %{HTTP_HOST}!^domain\.com 
RewriteRule (.*) http://domain.com/$1 [R=permanent,L]

- Ash

[Martin Dunst]

googleguy,

first of all, thank you for all the hints you give here at webmasterworld.

would returning a "410 Gone" status code be appropriate if the requested resource is gone for good?
i know googlebot is sending http/1.0 requests, still i wonder how it would deal with a http/1.1 (i.e. 410) response code.

regards
martin

[nolen1]

Thank you hetzeld and anallawalla for your suggestions but I have tried both and it still does nothing. I wish I knew what I was doing wrong. I have been trying to figure this out all day. I am using frontpage so when I saved this in notepad I saved it in my web. Is that right? And in cuteftp I deleted the .txt that was after the .htacess. Is that right?
Karen

[sabai]

The filename should be exactly '.htaccess' - not .htacess - i.e two c's and no .txt at the end. This is only for apache with mod rewrite installed... not IIS for example.

[AthlonInside]

.htaccess work for me. Just 1 update, the non www domain is removed from the index.

However, where will be an update? Is there anymore?

[nolen1]

I'm sorry for the mistake. I have been using two c's. My hosting company is Doteasy they told me that I could use a .htaccess file. They said to make it in notepad and upload with cuteftp but I still can't get it to work.
Karen

[hetzeld]

Hi Karen,

When transfering your .htaccess file via ftp, make sure to select an ASCII transfer and not a binary one

If you use any of the rewriterules, you'll need to activate mod_rewrite with:
RewriteEngine on

Dan

[anallawalla]

It is possible that your host has not installed all the modules needed for .htaccess to work fully, e.g. at another web host I can do a 301 but I cannot do a rewrite. I can't think of the name of the other forum on this site but with a site search you'll find many good discussions about .htaccess there.

- Ash

[Kukenan]

Thank you GG
I will patiently wait for the 301 to do its magic.

I just realized that Google has my homepage under www.site.com and with fresh cache (from july 24th).
So I guess it is starting to work.

Thank you guys.

Great forum!

[nolen1]

Thanks again everyone for your suggestions but no matter what I do it won't work. I guess I'm going to email my hosting company and ask them why. I doubt it will help since I did it the way they told me. They told me I would need to sign up for frontpage extensions so I did. Is that true? I hate to say it but if they don't have any answers I guess I'll just have to give up and hope Google can straighten it out. If I moved to a different hosting company it would probably cause even more problems. I don't understand why this problem exist. Is there a way a person could really have two different sites at http:// and [www?...]
Thanks again
Karen

[hetzeld]

Karen,

mod_rewrite is IN NO WAY related to FrontPage extensions .
If this is the answer you've got, maybe your question wasn't clear enough :(
When modifying the .htaccess file, be careful with all the stuff in there used for the Frontpage extensions. Don't erase them but add the rewriterules at the end of the file.

Good luck!

Dan

[nolen1]

Dan,
I told them my problem and I asked if I could make a .htaccess file and include this:

RewriteEngine on
RewriteCond %{HTTP_HOST}!^www\.domain\.com
RewriteRule ^(.*)$ [domain.com...] [R=301,L]

They said I would need frontpage extensions and they reccommended using notepad and cuteftp. Is there anything else I should ask or tell them?

Karen