Welcome to WebmasterWorld Guest from 54.224.230.193

Forum Moderators: goodroi

Message Too Old, No Replies

Google showing site despite blocking in robots.txt

The page is not cached though

     
4:20 pm on Nov 20, 2007 (gmt 0)

Preferred Member

5+ Year Member

joined:Sept 28, 2007
posts:487
votes: 0


I do not want Google to index a recently launched website and followed its guidelines. However, in the search results I see a listing like this:
www.domain.com
Similar pages
There is no title or description or cache or file size.
Is this what I should have expected?
4:25 pm on Nov 20, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts: 12169
votes: 55


Yes, this is normal behavior. It is referred to as a URI Only Listing and occurs in this scenario. No other pages from the site will show. Just the root level reference.
4:28 pm on Nov 20, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 29, 2002
posts:1954
votes: 0


yep, as long as they have already indexed the page(s). If you do not want the pages indexed you need to put one of the following in between the page(s) head tags:

<meta name="robots" content="none">

or

<meta name="robots" content="noindex,nofollow">

and then change your robots.txt to re-allow them to get to the page.