cpollett - 8:16 pm on Apr 1, 2012 (gmt 0)
So I left open in my robots.txt that google could still download statistics pages just not follow any links off of them. The refresh rate on robots.txt as mentioned is probably less than 24 hours (it's actually easier to make your crawler faster if you check robots.txt not so infrequently because it means you don't have to keep as big a cache of them in memory). I am still getting requests from google for the nonexistent stats page. The reason is that I have a token for anti-CSRF that I tack into the query string. So Google, presumably while there were still links to that page, was extracting a new link each time it was requesting one of the other pages in my site. So the fact that I am still seeing requests to that page, each with a different token suggests to me there queue takes at least several days from link extraction to actual download.