On a couple of my sites most of the referrals are from image searches even though there is plenty of text. I would say they are penalized. Here's one site's Referrer Report.
reqs: site
----: ----
29: images.google.com/
29: images.search.yahoo.com/
28: search.yahoo.com/
28: search.msn.com/
25: www.google.com/
12: www.altavista.com/
10: pictures.ask.com/
"site:mydomain.com" search on Yahoo shows 487 pages.
"site:mydomain.com" search on Yahoo images shows 238 images.
"site:mydomain.com" search on Google shows 304 pages.
"site:mydomain.com" search on Google images shows 142 images.
Here's another site's Referrer Report.
reqs: site
----: ----
71: search.msn.com/
30: images.google.com/
8: images.google.fr/
7: images.google.it/
7: images.google.com.mx/
5: images.google.co.uk/
5: images.google.ca/
4: search.msn.it/
4: www.google.com/
4: images.google.nl/
"site:mydomain.com" search on Yahoo for this site shows 0 pages.
"site:mydomain.com" search on Yahoo images shows 0 images.
"site:mydomain.com" search on Google shows 20,200 pages.
"site:mydomain.com" search on Google images shows 543 images.
Yahoo and Google trust the sites are giving the correct image even though they don't trust the text.
