aakk9999 - 1:25 pm on Jun 28, 2013 (gmt 0)
My question - is there any differential treatment for password protected pages vs robot.txt excluded pages in Google SERPS? We know that robots.txt excluded pages show up as link only stubs in SERPS with a description posted by the OP. But what about password protected pages? If they don't show up in SERPS at all, why this differential treatment?
Yes, there is a different treatment.
As others said above - if the page is excluded from crawling via robots.txt, Google is only told it is not allowed to crawl the page and therefore will not be (should not be) requesting it. Hence it cannot see any other directive such as:
- HTTP response code (including these that are 301, 401, 403, 404, etc)
- on-page robots meta such as noindex, noodp etc
Hence pages that are roboted out may show in SERPs as Google was only told not to crawl them and does not know about any other directive or response code that might result in different page handling.
Think of it like this:
If I forbid you to ring the doorbell, you cannot tell whether I am at home or not, in fact you cannot even see if it was me living at this address nor even whether the door exists. All that you know is that someone talked about my door.