Hi geoline - Apparently you've been at WebmasterWorld for quite a while in lurking mode. Welcome to posting mode. I'm sorry, though, that your post is about a ranking problem. I've had no experience with the particular problem you describe, so I can only suggest some lines of thought that are all conjecture.
Normally I would question your use of the word "penalty", and I would say it might more likely be...
a) something more mechanical relating to your redirects or server setup (and I would still check out those things first)...
b) or that you'd simply come to the end of the "honeymoon period" that Google gives a new domain....
c) or that changing your domain caused Google to reexamine your site, and now you've got to re-establish trust...
etc.
Since you've used WMT to report the domain change, and because of your long ranking history, these aren't likely to be the case, but I can't be sure.
Let me explore one odd algorithmic scenario that comes to mind, which might make this perhaps something closer to a penalty...
You say...
I changed the domain name from a keyword-domain to a more brandable name
Complete speculation here... but it may be that removing that keyword domain, particularly if it was a hypenated
keyword1-keyword2 domain, might shift your inbound link anchor text from being "natural" with regard to your previous domain to perhaps looking manipulative. It is sometimes a spammer's trick to promote hyphenated keyword domains to get anchor text links and then 301 these to a more natural looking domain.
Generally, though... and again, this is speculation... I'd think this would be considered spam only if you did this with numerous domains all pointing to the same domain, all with a lot of exact match anchor text... and/or if you'd built a lot of links quickly and then did the redirect.
It may also be that changing the domain name changed enough content and titles on your site that you need to add some of the keyword content back to your pages. This may not be seen so much as spam but rather more as a mismatch between inbound links and onpage content. You may be effectively underoptimized onpage and over optimized in your inbound anchor. What kind of content changes did you make?
If Google has reexamined your site because of the domain change, it could be that link acquisition patterns previously unnoticed might be seen as unnatural. Again, all conjecture, but perhaps this might trigger some thoughts for you which you can share with us.
Having more exact dates on the drop, etc, would help check whether algorithmic changes were observed around the time of your change.
