dstiles - 9:54 pm on Nov 16, 2010 (gmt 0)
Google is also not playing fair with their preview bot. It comes into my server across all sites through googlebot IPs that have a proper rDNS and on IPs that have no rDNS at all. "We will always run our bots with reverse lookup available so you can check it's genuine". Yeah, right, you and Bing both!
How do we know preview is a genuine bot? It's easily forged and a fair few of google's IPs are used by the general public or (often worse) by apps creators, some of whom have been proven to be hackers or even criminals. To display Previews of our sites on google's now-corrupted SERPS we have no obvious recourse but to allow ANY google IP onto our servers providing it carries the web preview UA. There is no way of checking its legitimacy.
And what about images disallowed in robots.txt? Some of our sites show images culled (by the preview bot?) illegally (disallow images directory in robots.txt), others show no images at all - one site even shows as "not available" (and contrary to one suggestion, it is a school site and certainly not a #*$! site!).
I was read one report which suggested that Previews could be scraped from SERPS - easily enough done - and used on other web sites. Now that's going to be intersting!
And on top of this is the copyright issue, which is not trivial!