Samizdata - 1:53 am on Dec 28, 2011 (gmt 0)
some of those are Bing
And they don't necessarily have "Bot" or "bot" in the UA (neither does Google Web Preview).
My point was that since the introduction of Google Web Preview, the lip-service formerly paid to robots.txt compliance by major search engines seems to have been deprecated.
Compliance was always voluntary, now they are volunteering to take anything they want.
There is no law against it, after all.