Now, if your opinion is correct ... why? The implication here is that after March '04, a site could be hurt (not penalized, hurt) simply by improving its design and navigation structure. That, to me, is a colossal failure on Google's part.
I don't know how to reply to the "especially" part because I suspect we all have different definitions of conservative SEO. :) The site in question is long on quality content, and short on IBLs. The redesign was done, frankly, with both users and SEs in mind -- we lightened page size substantially for faster loading, graphical buttons were replaced with text for minor added internal anchor text help, the 3rd-level pages were shortened to 2nd level, etc. Seems conservative to me, but I just don't get why changes like this would negatively impact a site that's been online since 1999. As I said, I consider it a failure on G's part.