There's no doubt that Facebook Connect gives a great deal of benefits, saving time logging into many different sites.
The down side is that you will be logging in with the Facebook id (your real name). For some, that may be a privacy concern. In addition, the activity you partake in at those other sites will appear on your Facebook account. Again, some may see that as a privacy issue.
Finally, my biggest concern is over the security of the username and password. Clearly, if you don't take great care of this, and perhaps utilise a high strength password, or it gets hacked, your identity may be compromised in more than one location.
I guess this also applies to any open ID, such as Google.
I have the same concern with the whole OpenID schema. It all started out with Microsoft's Passport IDs. MS IDs used throughout their entire network really bugged me in the beginning and I now have a bunch of Live IDs.
These Google and FaceBook Connect programs have me a bit leery. There are some things I don't mind having connected to my SNS IDs, but for the most part I'd prefer a separate ID and password for each.