dstiles - 8:34 pm on Nov 18, 2010 (gmt 0)
I'm not sure it IS smarter, Bill. SOME of our sites have previews with a full set of pics and others don't. All have the images and pics folders disallowed in robots.txt. It MAY be possible that the preview bot scraped them before I got around to completely blocking it but that's a lot of pages - and yes, I know your take on whitelisting! :)
On one of their pages google says it's possible to allow pics in robots.txt and then disable them per-page using the X-Robots-Tag meta, but I can't see that working and there are no examples. And, of course, that would mean feeding google a different robots.txt than all other SEs.
I really wish I could block google now! :(