tangor - 8:30 pm on Jan 16, 2012 (gmt 0)

I've been seeing the same thing in recent years... put that down to the sheer number of distributed IPs they are using. Seems a mish mash failure for the back end to communicate with the front line crawlers.

But, since my robots.txt is only a few hundred bytes (I only allow five in, all others out) it has not been an issue.