renaming and removing pages, custom 404s, 301s

and possible adverse effects in google?

[jamie]

in the 4 years we have been online we have renamed and reshuffled a number of pages in our large site.

* sometimes by restructuring the site to reduce clicks,
* sometimes taking down clients
* sometimes renaming directories more logically

we have always used a custom 404, a meta refresh for important renames, and more recently 301 redirects.

we have never known any adverse effects and as far as we can see google removes these nonexistant pages from its index and adds the news ones.

however, might too many of these missing pages might have a detrimental effect on our ranking?

* an example:
we have just taken down two client sites with 12 pages each. googlebot spiders our site and finds 24 missing pages and lots of 404s.

can we be penalised for this?

thanks for feedback

[Brett_Tabke]

It is one that no one could really know other than Google. It's not something you could test reliably.

I don't see what Google could gain by degrading sites with an extra amount of 404's. Freshbot should address it all.

[jamie]

thanks for feedback brett.

i am sure many webmasters are in the same boat, and i have never heard of any problems.

the speed with which google normally finds the re-locatd pages is excellent. so will carry on playing by the book with 404s and 301s

[jdMorgan]

jamie,

You might want to consider returning 410-Gone status when you take down a page for which you can provide no logical replacement or alternative. That's slightly "cleaner" and lets everyone know it's not a temporary error on the site - the page is, in fact, gone.

Where possible, I like to eliminate all 404's due to ongoing site maintenance leaving only 404's due to true site errors. This preserves the 404 log entries' usefulness as a true error indicator. 301, 302, and 410 status codes are helpful in this respect.

Jim

[jamie]

Jim,

that sounds even better. i've just had search for that but can not find how i can get the server to return the correct code?

obviously i could set it up in my .htaccess

ErrorDocument 410 /410.html

(with a bit of text on the 410.html explaining that the page no longer exists. Please go to the home page)

but how does the server know which is which?

thanks for help!

[jdMorgan]

jamie,

That depends on how you want to set it up. If you want to declare a resource (page or file) gone, you can use

RewwriteRule ^SomeRemovedPage\.html$ - [G]

Or you could declare a group of files or an entire subdirectory to be gone using a similar rule.

This approach can get unwieldy if a large number of resources are removed from the site. An alternative is to detect all missing files and return a 410 for them, but I don't recommend it. This technique again loses the distinction between "gone, and we know it's gone" and "missing because of a problem on our site."

However, to illustrate, you could use:

RewriteCond %{REQUEST_FILENAME} !-U
RewriteRule ^spring\_specials/[^\.]\.html$ - [G]

This returns a 410-Gone for any requested html page in the subdirectory "spring_specials" which does not exist.
Note that this is not very efficient in a .htaccess context; the RewriteRule should be made as specific as possible to avoid impacting the server on a busy site. The -U check-for-file-exists is the slow part. However, because of the way that mod_rewrite works, the RewriteCond will not be tested if the RewriteRule fails, so it is good to make the RewriteRule as specific as possible.

Perhaps a mixture of the two methods above would be workable for you. However, I recommend designing your approach with maintenance in mind; Usually, you can remove the removed-resource 410 redirects after 6-12 months when all the important search engines have recognized them, so it's a good idea to make that easy to do.

Tutorial & links to more info: Introduction to mod_rewrite [webmasterworld.com]

HTH,
Jim

[Craig_F]

301 question: We had an old version of our site running and just took it down and did a 301 permanently moved redirect to the new domain. Will the links to the old site now be transferred to the new domain, or do we need to do something else?

[jdMorgan]

Craig_F,

Wherever possible, you should ask other sites which link to your old domain to update their links. Other than that, leave the redirects in place long enough for the majority of your customers to update their bookmarks and for the other sites to update their links to you. When to pull the plug on the old domain is up to you, but it's a good idea not to "orphan" important incoming links.

Jim

[jamie]

many thanks for an excellent explanation Jim,

cheers

jamie