tedster - 12:17 am on Jun 1, 2010 (gmt 0)
A doctype applies to a document -- that is, to a "page" and not to a "website". There are many sites that mix doctypes with no trouble from browsers or search engines. In fact, it's REALLY common to see the HTML 4.01 doctype begin the soruce code but then some XHTML tagging thrown in. That doesn't make trouble and it is much crazier than having some URLs on a domain using a different doctype.
I have some frameset doctypes on sites that are mostly html 4.01 strict - not a problem and not invalid, ether.
So beginning to write new pages in HTML 5 without first converting the old pages is not going to make any trouble, technically or with the search engines.
It's an interesting fact that there is no technical definition for either the word "page" or the word "website", even though we toss them both around all the time.