i've been to both, they're both great!
at PubCon, you'll find more seo experts, and in-house people who do sem and webmastering of all kinds, definitely more diverse positions / jobs / skills. the sessions will have a wider scope as a result. not to say they're fluffy, they are focused and very high quality, they just cover more ground than Summit, because the audience is more varied and not constrained to an aff focus.
at Aff Summ, obviously, more focus on affiliate activities, and great networking specifically centered on aff-merchant relationships. there's great diversity here too, affs and merchants of all flavors, and far fewer employee types. nice breadth to their sessions too, often ranging outside pure aff focus, but still more narrowly focused on aff activities than PubCon is, for sure.
networking as an aff, doesn't just mean knowing affs, so don't dismiss PubCon just because your focus is mainly as an affiliate.
between hotel and airfare and conf fees, it adds up - your travel costs and timing may be a big factor, given how good both conferences are.
one guarantee i can make, these shows are fun! if you work in this sphere, you will be amazed at how open people are to discussions that will interest you, how great networking is, and how extensive the partying is - it's our chance to socialize and cut loose - and we do that in spades!