aakk9999 - 6:49 pm on Jun 28, 2013 (gmt 0)
But doesn't the same hold true for password protected pages? By password protection we are telling them, they are not allowed to crawl.
I can see what you are thinking but I believe it is not the same. If a page is password protected, then visitors cannot see the page either unless they know the password. Visitors that know the password may be only a selected few - and if a visitor knows the password, they would probably know the URL too. Hence it is probably not good for Google to have such page in index because if a click from SERPs requires a password it is most likely a bad experience for visitors coming from SERPs.
But restricting access via robots.txt is for bots only - they are not allowed to go there, but visitors see the page.
Unless you are cloaking, of course, and have a page password protected for bots only - then visitors would see it without password, but Google would not know this, so why should it show in its index.
<added>Which is pretty much what phranque said in the post above.</added>