would you trust google to be able to tell the difference?
these plugin backlinks generally result in a sitewide link in the blog sidebar. or a sitewide link somewhere on the template... on hundreds of different blogs.
one thing that i have noticed from looking through my logs, is that the plugin ends up on a lot of spammy auto-blogs, which are just looking for content. there's not much you can do about that, because once you've uploaded it to the wordpress directory you have given up some of your control. anyone can download and install it.
and of course its those spammy auto-blog things that are the most difficult to contact, to get the plugin removed.
it wouldn't matter if there's no backlink on it, and no mention of your site, but then what's the point of creating it?
the more i think about it, the more i'm put off doing wordpress plugins.