Forum Moderators: open
I've seen an SEO pack for Joomla, but I forget the name (I don't think it's the one hanyaz mentioned, but another one). If you hunt around you should be able to find a joomla add-on that helps you sort out the most pressing weaknesses.
The other main issue is that most CMS tend to have significant code bloat for a a variety of reasons.
Personally, I just can't be bothered to code from scratch anymore for a simple content site. The positives just way outweigh the negatives in my mind.
As others have already said, it does require a bit of tweaking, but once you have one site tweaked properly, it's pretty easy to use that as a standard template for other sites.
There are a number of SEF modules that work well. Hanyaz mentioned one, another I've used with good success is SEF Advance. Both do close to the same things.
Another addon is called Joomla SEO Patch which is free which helps to structure some of the code a bit better and adds some additional SEO functionality not found in the core joomla system.
As ergophobe mentioned, code bloat is a definite issue, but the speed at which you can put up and the ease with which you can maintain CMS based sites definitely outweighs the negatives.
The only other issue is security. Obviously, a CMS has a number of security issues that you won't deal with in a regular static html site. Properly securing the CMS from the beginning and keeping up with security updates is extremely important or you will have lot's of other issues to deal with.