Sure. In a couple of cases, I was cheated outright. (The merchants simply didn't pay, even when their own statistics showed that I was owed a fair-sized chunk of money.)
More often, the disappointment has come from a mismatch between what the affiliate program has been selling and what my audience has wanted. It takes time and experience to learn what works and what doesn't, in terms of both (1) selecting affiliate partners and (2) integrating them into the site.
Disclaimer: My comments apply to editorial "content sites," where the affiliate typically needs to make affiliate programs work with an existing target audience. Someone creating a pure-play affiliate site is likely to have different opportunities (such as a having a blank slate going in) and different challenges (e.g., attracting traffic).