Google showing site despite blocking in robots.txt
The page is not cached though
4:20 pm on Nov 20, 2007 (gmt 0)
I do not want Google to index a recently launched website and followed its guidelines. However, in the search results I see a listing like this: www.domain.com Similar pages There is no title or description or cache or file size. Is this what I should have expected?
4:25 pm on Nov 20, 2007 (gmt 0)
Yes, this is normal behavior. It is referred to as a URI Only Listing and occurs in this scenario. No other pages from the site will show. Just the root level reference.
4:28 pm on Nov 20, 2007 (gmt 0)
yep, as long as they have already indexed the page(s). If you do not want the pages indexed you need to put one of the following in between the page(s) head tags:
<meta name="robots" content="none">
<meta name="robots" content="noindex,nofollow">
and then change your robots.txt to re-allow them to get to the page.