Page is a not externally linkable
PaulPA - 4:31 pm on Jan 26, 2006 (gmt 0)
While the ruling may suggest that a website operator makes an effort to block cache from happening, it also places requirements on search engines (and others crawlers) to show that they will abide by the robots.txt. That could suggest that some level of standardization for all crawlers may be coming.