Google announced the Page Layout algo update back in early 2012 and some webmasters who served more ads than content reported a drop in rankings. From what I saw around that time, you had to seriously screw up visitor experience to be affected.
Also, a site can suffer a drop in mobile traffic if it shows popups/unders that affect visitor experience on small screens.
However, 80%, sounds like a huge drop in traffic and my gut feeling is that it's not the pop-under that has caused the drop.
Is the pop-under serving "safe" content? I mean, has it been checked for viruses and malware? Have you tried to "fetch as Googlebot" to confirm that the pop-under doesn't affect the way Google sees your pages?
Have you pinpointed the date the drop happened? Did it drop like a stone or was it a gradual decline?