This won't come as any surprise for Google. And don't forget, Google has a huge head start over Yahoo/Overture. It owns the mom-and-pop publisher marketplace, and that's unlikely to change soon.
The biggest danger to AdSense is the possibility that Yahoo could skim off the cream of its network. If the Yahoo folks are smart, they'll recognize that AdSense's biggest strength (virtually uncontrolled growth) is also its biggest weakness. They'll have measures in place to avoid the problems that Google has had with scraper and other junk sites, and they'll position themselves as a premium service for advertisers who aren't willing to take potluck.
Mind you, there are several "ifs" here; one being how Yahoo goes after Google, and another being how Google responds. For all we know, Google may have its backup game plan sitting in a safe somewhere, waiting to upstage Yahoo. That would make a lot of sense: Let Yahoo come into the marketplace with trumpet fanfares, then announce "Um...we've got something better" and roll out AdSense: The Sequel.