Page is a not externally linkable
loudspeaker - 4:55 pm on Sep 30, 2007 (gmt 0)
What if they are happy to receive Google results from Google the-search-engine but not too happy to see their snippets and photos in Google the-restaurant-review-aggregator? Is there any way for them to allow one and disallow the other? (The way it stands, I don't think so) Do you think there should be a way? Again, I don't necessarily think Google is the worst offender here - as you pointed out, at least they give you links. But don't you think there's a substitution of the model going on here? The old model worked fine with robots.txt. The new one seems to need another config file. Simply referring content authors back to robots.txt is essentially trying to push a "packaged deal" on them (either no remixing, but also no search engine traffic or you get search engine traffic, but your content may be used any way the search engine likes).
I'd imagine that most publishers of reviews, etc. are happy to get the additional traffic, and that they'd ban Googlebot with robots.txt if they weren't.