Googlebot and indexing (404 errors)

404 page error, spider, actions

[cl328]

Hello,

I have recently (last weekend) change my site to a new directory structure. Well, I also moved web servers too.

Anyway, the site gets a ton of googlebot traffic (it was a message board). The site has many visitors coming from Google. Google has hundreds of pages indexed in the search engine.

Now, googlebot is going to these old/incorrect links and/or pages. They are incorrect because I have changed the directory structure of the whole site.

My question is:
Googlebot is going to these wrong links, the return code is 404. (although I have a custom 404 html page, basically a "welcome to the new website announcment page" that have a link to the new homepage)

Should I worried about the googlebot indexing wrong pages and getting the 404 code?
Should I create a similar directory structure, just to prevent the bot from getting a 404 error?
Will google eventually un-index my website pages?

Google has slowly indexing my new site, BUT I'm afraid of loosing a LOT of links and traffic.

what do you think?

Thanks

[Netizen]

Do you have any easy way of knowing how an old page maps to a new page? If so you should implement a 301 redirect from the old pages to the new pages - Google is trying to index pages on your site that is have found in links elsewhere on the web. If you want to keep those as backlinks then you need to start redirecting Googlebot to the new pages.

[mack]

In Theory Google will index your new pages and remove your old pages frm the index. When googlebot finds a page that resolvs as a 404 it will remove this page from the index when it next updates. However if Google has found your index page it will begin to crawl your new directories. It will find the new content but it may take time. It would speed up the process if you where to use htaccess redirection. This is however not always easy if the directory structure is very different.

What you may want to do is make all pages on "old folder" point to index page in "new folder" This method isn't perfect but may help.

Hope this helps.

Mack.

[takagi]

although I have a custom 404 html page, basically a "welcome to the new website announcment page" that have a link to the new homepage

Maybe you should check if the custom 404-page is really giving a 404 using the Server Header Check [webmasterworld.com]. Enter an old URL and check if you see something like:

HTTP/1.1 404 Not Found
Date: Mon, 18 Aug 2003 12:38:42 GMT
Server: ....
Connection: close
Content-Type: text/html

[cl328]

Thanks for replying...

I'm a little confused about the redirect stuff.

I did the server head check and this was the results:

HTTP/1.1 200 OK
Date: Mon, 18 Aug 2003 13:24:51 GMT
Server: tigershark/3.0.102 (dn1.directnic.com)
Connection: close
Content-Type: text/html
Content-Length: 349

Is that good or bad?

I guess since the directly structure has changed, I will lose all those back links?

Unless, I recreate the directly structure again....Which is just to much to create.
:(

thanks

[Slade]

If your custom 404 is actually handing back a 200, Googlebot is seeing the custom 404 page as the new/current content for each missing page on your site.

If you don't want to lose the PR or indexing you have, I would start building a 301 list into your .htaccess telling the world where to find your pages.

If you don't want to recreate the old structure, start emailing everyone that has deep-linked to you and ask (nicely) that they update their links.

You won't get them all, and you won't find everyone with old bookmarks but that's what happens when you make large changes to directory structure.

[Netizen]

Returning a "200 OK" for the 404 page is not good as Google will see that as a real page as opposed to an "oops, that page isn't here any more" page.

The tedious answer is to track down the backlinks and get them to amend them to the new locations.