---- page is noindexed, but still shows in SERP with a Google notice
lucy24 - 10:46 am on Jun 30, 2013 (gmt 0)
147 times while crawling 531K pages.
You can't set a per-page quota. Well-behaved small robots pick up robots.txt at the start of each separate visit. Large robots-- and you can hardly get bigger than the googlebot-- read robots.txt, spread it around to their fellow googlebots, and hold it for up to 24 hours.
As I understand it, "nofollow" doesn't mean "pretend you haven't seen this link". It just means "I make no claims about the quality of the material I'm linking to".