To consider several kinds of problems that might be affecting you in Google, but not in Bing or Yahoo....
last year I accidently added some spammy directories which i removed later on.
I'm not clear what you mean by "added some spammy directories". What was spammy about them? How long were they up? How long ago did you remove them? In the process, did you make other changes to your site as well?
I'm wondering whether you might have done something which might have appeared to Google that you tried something spammy and when you saw it didn't work you changed back. Google's got a patent on a penalty for that. It's not certain they've implemented it, but I'm thinking that they have. See this discussion.... Google's Rank Modifying Patent for Spam Detection http://www.webmasterworld.com/google/4486158.htm
More likely, though... as Google has been applying refinements to its algorithm, it is raising the bar, and it may be that your rankings are simply eroding because your content is no longer special enough and your backlink quality isn't high enough.
How are you getting your backlinks?
"Blogs and press releases", which you mentioned, have also gone through a bunch of re-evaluations by Google as inbound link sources. If press releases aren't rewritten into unique articles, chances are that they aren't helping you. If you've included anchor text in them, they might be hurting.
If you're guest blogging for links, you might also be running into problems. Google might be considering them self-placed links and treating them as spam. Please elaborate what you're doing.
These are the kinds of things that might be affecting your Google rankings, but not Bing or Yahoo.