OK, all of the sites (ours) that went AWOL are back after 5 days this time, it was scary (though, I am getting used to this pattern now) when authority years old sites are treated like that. So, they disappeared on March 15th for almost 4 days, back with 20% improvement on most. Disappeared for 5 days on 16th April, back on the 22nd April (close to 6 days actually if you count all in hours) so far with 15% improvement, I am expecting or hoping for 30-40% improvement, because that heavy deep crawling probably wasn't just indexing more pages BUT having a second thought about sandboxed pages and checking for valid URLs. The heavy crawling is still going on, though easing a bit.
Looks like in most cases, and further to a collective opinion to which I seconded mine, when sites disappear for few days, heavy deep crawling starts going on for those few days, then those sites are back with improved positions and SERPs. The question as to why they have to disappear is, and as I suggested before, is to do with the switch between the old and new structure, further to that, it looks like one of the two or both is happening as well, filtering / sandboxing and /or de-sandboxing and / or SERPs adding / improving to those sites as well.
Now, few days maybe for very authoritative sites, such as WebmasterWorld or at least old established and Alexa top 100k sites, for others it may be weeks!
I am seeing a massive increase in gbot activity as well and been hit with a big downturn in traffic to the one site that is now getting massively crawled.
If I am right, and providing you sites are white hat, oldish and have some authority and mostly original / unique content updated at least weekly (or eternally useful content which does not need any updating), they will be back before the end of the month with some improvements.
dusky now that you said that yes I do remember that post. I think it was about the time we were getting all the updates on each and every ip Google owns where they could see Caffeine results and it was buried so fast it just didn't sink in.
Yes, I think they are still doing that, testing live likely and probably on the old DCs NOT the caffeine DCs this time, the consensus now is 70%+ IPs are caffeinated at least for North American audiences and 50% for Europe and the rest of the world, is a good reason to do the testing on older DCs.