I took a quick look at your site in your profile. Looks like a penalty to me when there are no backward links showing in Google when there are 160 showing in Alltheweb. I'd look at these factors:
1) Taking out the link to the off theme sites, especially the one to the site with the other popular, but more-inclined-to-be-Google-bashing, SEO forum.
2) Inbound and outbound links - very optimized to maximize and preserve index page Pagerank. Maybe too optimized? I also noticed extensive cross linking between this site and another, very similar site, that also has your name in it.
3) Inbound links from at leat one gambling site, at least one Zeus site, other off topic sites, including at least guestbook link on a totally unrelated topic.
These are all things I'd red flag if I was writing the code for a SE.