This question has been debated for years. Certainly if you have a site that consists of little but affiliate links you are unlikely to rank well in Google, etc. Is that because you are being penalized or because you aren't getting links TO you from established sites? Possibly both.
Google has or had a program for looking for "thin affiliates" and penalizing them--Google that phrase and you will find out more.