Pfui - 3:00 pm on Sep 3, 2011 (gmt 0)

The question is will they obey webmasters and not crawl what we say don't crawl. The answer appears to be no.

That's been true a long time, unfortunately. You'll find scores of reports in WW's "Search Engine Spider and User Agent Identification [webmasterworld.com]" forum.

For example, here's fresh info about non-obvious Twitter-mining:

Resolving "urlresolver" | Google IPs repeat no-robots runs
Recap post: [webmasterworld.com...]

And more GWT news:

Google Web Preview | Not just from bare IPs anymore... [webmasterworld.com...]

After spending too much unrecompensed time 'accommodating' GWT before G worked out their own bugs, I will no longer kick their tires for them via +1 or anything else. I'm seriously weary, and increasingly wary, of jumping through their we-cloak-but-you-can't hoops.