How to handle a large renamed subdirectory

[beggers]

Let's say I have a site with a huge subdirectory of files that have been heavily spidered by Google. It looks like this:

www.examplr.com/123

That entire subdir of files bacame obsolete do I had to create a new subdirectory to install a complete new set of upgraded files (I couldn't use the old directory name):

www.example.com/456

The thing is, I want Google to stop spidering /123 and start spidering /456.

So the first thing I did was to rename the old directory to something like /123-OLD (because I don't want those files found anymore).

I also made a custom error page that redirects all the "file not found" links to the /456 index page.

Is this a good or bad way to get Google to start spidering the new directory? Any suggestions?

[edited by: tedster at 7:42 am (utc) on Feb. 1, 2008]
[edit reason] use example.com - it can never be owned [/edit]

[Marcia]

You could also do a 410 for /123 with a custom ErrorDocument that will send users to the new /456 directory. Are the filenames the same as they used to be?

[beggers]

No, it's a new set of data - the links don't match at all.

[tedster]

I want Google to stop spidering /123

If you place that directory in a robots.txt Disallow rule, that will stop the spidering. Otherwise googlebot will continue to request it indefinitely.

However, if the old directory has any significant backlinks, then you may be squandering link juice and PageRank with that approach. So if the old directory does have decent backlinks, you may want to use a 301 redirect to the updated content.

If you can't easily create a page-by-page redirect, then at least redirect requests for urls in the old directory to the new directory's index page. But getting more detailed and creating 301 redirects for just those key urls that have external backlinks will take this to the next level for you.

I also made a custom error page that redirects all the "file not found" links to the /456 index page.

If the http status is 301, this sounds something like what I'm talking about. But do be careful with that status code. Even a 302 Temporary Redirect is not what you want.

rename the old directory to something like /123-OLD (because I don't want those files found anymore).

Be sure to place a Disallow rule in your robots.txt for this renamed directory. Google has ways to find files that we think are hidden and not linked to. I would even suggest removing that directory from your server altogether. Then even the "bad bots" won't index it.

I want Google to... start spidering /456

Make sure you've got some links pointing to that directory, and a decent structure of internal links within the new directory. Google will then follow that link structure and index the urls in your new directory.

[beggers]

Thanks, tedster, that was really useful information!