Pfui - 3:00 pm on Sep 3, 2011 (gmt 0)
The question is will they obey webmasters and not crawl what we say don't crawl. The answer appears to be no.
That's been true a long time, unfortunately. You'll find scores of reports in WW's "Search Engine Spider and User Agent Identification [webmasterworld.com]" forum.
For example, here's fresh info about non-obvious Twitter-mining:
Resolving "urlresolver" | Google IPs repeat no-robots runs
Recap post: [webmasterworld.com...]
And more GWT news:
Google Web Preview | Not just from bare IPs anymore... [webmasterworld.com...]
After spending too much unrecompensed time 'accommodating' GWT before G worked out their own bugs, I will no longer kick their tires for them via +1 or anything else. I'm seriously weary, and increasingly wary, of jumping through their we-cloak-but-you-can't hoops.