kheadley - 8:38 am on Oct 5, 2012 (gmt 0)
One that I found quite interesting;
#82407. [project “Other Search Features”] For pages that we do not crawl because of robots.txt, we are usually unable to generate a snippet for users to preview what's on the page. This change added a replacement snippet that explains that there's no description available because of robots.txt.
Does this mean that Google still indexes pages that have been disallowed by the robots.txt perfect example search "terms and conditions" and find the result;
Terms and Conditions - The Times
A description for this result is not available because of this site's robots.txt – learn more
This page has been blocked by robots.txt but is still indexed?