Just to keep this going.
We did find an affiliate that was linking to us from adult content .. via of course affiliate link, with affiliate ID attached. What I originally thought was a competitor, turned out to be an affiliate. We've reached out to that person and they removed their links from adult sites. We terminated affiliate agreement with her and waiting for our banners to be removed from all her other regular sites. Just hoping she complies.
Clearly a penalty, clearly Google should be able to recognize that no serious business would resort to link schemes like that, and just discount these.