I'm looking at the lack of growth in AdSense from a broader scope, the web is simply changing.
First of all, the dropping of AdSense's share in Google's total revenue is simply due to the fact that Google's search ads grow faster than AdSense's display ads. This is no news, search always had an upperhand in targeting. What is alarming is not AdSense's drop in share, but its own lack of growth, which is currently around 5% YoY.
The correct comparison is not AdSense vs Google owned sites, but AdSense vs the total display ads market. The correct question to ask is why AdSense only grows at 5% YoY while the total digital display ads market is growing at around 20% (US numbers, I suppose global numbers should be similar or even higher)?
Here are my speculations:
1. Traffic. I have no numbers as I can't find a study on this subject, but I suspect the web is more and more dominated by big players. I suspect the traffic share of the top 10 or 100 sites has slowly but steadily been growing over the years, which means that traffic to AdSense publishers is not growing as fast as the market. Below average traffic growth leads to below average revenue growth.
2. Relevence. Keyword-based contextual ads are getting old. Facebook can provide better targeting on display ads utilizing their social data, data which Google doesn't have (RIP Google+). In fact, AdSense's stagnation started in 2013 (it grew a healthy 20% in 2012), in the same year Facebook's display ad revenue grew 50% (vs 25% in 2012), a coincidence? On a side note, AdSense is still doing much better than Yahoo, whose display ad revenue shrinked 6.4% in 2013.
3. Google's actions. I figure Google may have perceived Facebook's threat to its display ad business a few years back, and has since been taking measures to improve AdSense's competitiveness. The current stagnation in growth may be a result of trying to get rid of the bad guys (and inevitably some good guys by accident), which may be harmful short term, but should be good for the long term. Wait a few years and we will see if this is the case.
I don't believe for a second that Google is abandoning AdSense. Display ads is still a big market, about as large as search ads, and it's currently experiencing rampant innovation in targeting techniques, so its future could be bright. Don't forget that even Google needs to diversify, it's not healthy to put all its bet in search ads. AdSense is Google's major weapon to fight in the display ad market.
What's more, Google can't really shift revenue from AdSense to its own content sites, they grow independently to each other. Yes, it probably can save some high CPC ads to themselves if the advertiser didn't specify where to put them, but in general I don't think AdSense publishers are in conflict with Google's own display ads properties like Youtube. If Google abandons AdSense, I suspect most of the revenue won't shift to Youtube, but rather to Facebook, Twitter, Yahoo, etc.
There are also a load of other benefits Google can get from AdSense, most of which are already mentioned in this thread. I'm sure Google is fighting to grow AdSense, although whether it will succeed is a totally different matter, I sure hope it will, but can't be too optimistic.
All in all, the web is changing, and the ad market is getting more and more competitive, the good old days when anyone could build a website and earn easy AdSense money is gone. It has been said a million times but you really can't take AdSense revenue for granted any more, instead you need to work hard to create a truly valuable website, gather loyal users, and think about how to best monetize it.