Not necessarily, if you are talking 50-90% of internal links go to .jpgs]
2 years ago I received a new client with PR0 penalty. After a while I found out that almost all images were "hijacked" from another site. This was done unintentionally (he was a distributor who simply wanted to use images from the main catalog) and still it took us almost 2 month to remove the penalty.
It`s always better to use your own assets.
BTW: Penguin is much harsher on hot-linking.
Nothing is "too small" that`s why it`s called "optimization".