Convergence - 2:34 pm on Apr 12, 2013 (gmt 0)
Hmmmm - the Google is having no problems crawling us at thousands of pages per hour. One would think if hosting providers were getting flooded that crawling/indexing would be affected.
As I mentioned before, our traffic pretty much stopped cold a week ago tonight at 11:04PM EST. Crawling by the Googlebot had been low up to that point. Next day, and since then, crawling has been very heavy - don't remember ever seeing crawling this heavy.
On the traffic we are getting from the Google, we are seeing less 404s. We suspect that the Google is cleaning up their SERPs. Now 95% of our traffic was never 404s, but 95% is about what our traffic dropped.
Also, we are seeing traffic to brands/pages that, in the past, received little to no traffic before. Traffic to our "bread and butter" brands/pages are what were hit. We have about 1M pages on this site. In the past week we have seen close to 30,000 new pages added - up 8,000 from yesterday. This site has over 300 sitemaps, each sitemap is updated a minimum of once a day. Once a category/section is updated the SEs are pinged. Bots come a crawling.
It's almost as if the Google is going through each sitemap, testing it for traffic, and moving on to the next.
Don't know - have never seen behavior quite like this...