Samizdata - 7:45 pm on Dec 27, 2011 (gmt 0)

The introduction of Google Web Preview sounded the death-knell for robots.txt compliance.

It is no consolation that Microsoft bots are even worse.

Only enforcible restrictions seem to work these days.

...