indyank - 3:25 pm on Jun 28, 2013 (gmt 0)
Hence pages that are roboted out may show in SERPs as Google was only told not to crawl them and does not know about any other directive or response code that might result in different page handling.
But doesn't the same hold true for password protected pages? By password protection we are telling them, they are not allowed to crawl. Google is forced to obey as they might not know to break past the password. But Google does get a hint a page exists for that URL as someone has linked to the password protected page. Why aren't they showing the password protected URLS in the SERPS with a boilerplate description, like they do for roboted out URLs?