Matt Cutts went on record as saying good spelling and grammer is not a ranking factor, and that the fact that well spelt sites rank is corralation, not causation. However much salt you want to take with that tidbit is up to you. I took it with a fair amount.
My 'gut' feelings - A lot of search users are part of online communities where that sort of 4Chan-esq language is used. Google would be throwing a lot of babies out with the bath water if they put in place a blanket 'Bad spelling = penalty' flag. If spelling and grammer is a ranking factor, its much more nuanced than that, and will be applied in conjunction with a whole host of other signals. I wouldn't worry about the comments from an SEO point of view.
Besides, if I were a black hatter and I wanted to get your site done for spam, I wouldn't be anywhere that subtle. I'd hit your site with 10,000 comments about Viagra and Boobies, keep hitting it faster than you can remove them with a bot, and point a load of crummy backlinks from the shadiest bits of the internet while I was at it.
Are there links being added with the comments? It could be a spam bot or agency with grammer like that.
If you've been hit with Panda (How do you know it's Panda, by the way? There are plenty of ways to get delisted from Google besides onsite content and design elements) I'd look elsewhere for the reason if I'm honest.