Affiliate and AdSense aren't mutually exclusive. You can have them both. How much of a distraction one is to the other is up to you.
AdSense doesn't require much effort on your part - you just set it and forget it. You are prohibited from encouraging anyone to look at it, much less click on it.
But with Affiliate offers, you get to do your own selling. In fact, it pretty much won't work unless you do. Very few people just whip out their credit cards because they saw a link or banner on your site; you have to warm them up first.
I use AdSense where I have a lot of traffic, but it's not entirely targeted enough (other than maybe by location) to the point that I could promote the proper affiliate offers. I've tried them, and they don't work as well here. Direct ads are also a good choice for this type of site, and I'm doing more of that now.
I use Affiliate on smaller sites, where there is less traffic, but it's more focused on a single theme or issue, and the people who come there want to find solutions for that issue. I might tuck an AdSense ad in a corner here or there, but for the most part, I only want the users to concentrate on the Aff offers.
For me it's kind of a whole different set of user behaviors that determine what I use where.