---- Googlebot getting caught in robots.txt spider trap
lucy24 - 1:41 am on Sep 6, 2011 (gmt 0)
Serving the right error codes for both planned and unplanned outages is something that few sites get completely right.
OK, now I'm trying to wrap my brain around the idea of having control over what gets served up during an unplanned, uhm, anything. Is there a definitive thread that explains it? "Error code" doesn't seem to be a fruitful search string ;) (16,600 hits-- constrained to this site-- goes beyond "fruitful" into "rotting on the ground". Squelch.)