Crawling/indexing behaviour observations

Recently changed sites appearing incorrectly in SERP.

[hooptey]

Recently my company switched from CF to PHP and restructured our directories on our webservers. Googlebot crawls us everyday but for some reason the SERP is still showing the old CFM pages, and we are getting a giant 404 errorlog from G. The bots have seen and indexed the new pages, right? so why can't I search for them...? Perhaps the bots on G are only checking to see if there is *any* info still at our addresses and the SERP are pulling from some previous indexing? It's funny because our CFM date code still works in the cached versions of the page... A site switched over from CFM in June shows a July date, but it's all CFM, not PHP! weird!

any thoughts?

[guddu]

I would suggest you to read previous articles on this. There is a lot of useful stuff at webmasterworld. Use the webmasterworld search for an answer.

From my personal experience,

Use 301 redirection, use of robots.txt and the google URL removal system to remove all the old CF pages. I had a same problem while changing filenames and folders for a website, and it was solved by using robots.txt and Google URL removal system.

For 404 errors, you may redirect the users to your home page or a customized error page.

Use robots.txt as

User-agent: *
Disallow: /directoryname
Disallow: /dirrectoryname/filename.extension
Disallow: /dirrectoryname/filename.extension

and place it in the root directory.

(* would disallow all search engine spiders to crawl the above mentioned directories and filenames)

This way you may remove all the old pages within 3-5 days of using Google URL removal system and similary, other search engines would drop these pages in next crawl.