homepage Welcome to WebmasterWorld Guest from 50.19.169.37
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Google showing site despite blocking in robots.txt
The page is not cached though
ecmedia




msg:3509872
 4:20 pm on Nov 20, 2007 (gmt 0)

I do not want Google to index a recently launched website and followed its guidelines. However, in the search results I see a listing like this:
www.domain.com
Similar pages
There is no title or description or cache or file size.
Is this what I should have expected?

 

pageoneresults




msg:3509875
 4:25 pm on Nov 20, 2007 (gmt 0)

Yes, this is normal behavior. It is referred to as a URI Only Listing and occurs in this scenario. No other pages from the site will show. Just the root level reference.

The Contractor




msg:3509880
 4:28 pm on Nov 20, 2007 (gmt 0)

yep, as long as they have already indexed the page(s). If you do not want the pages indexed you need to put one of the following in between the page(s) head tags:

<meta name="robots" content="none">

or

<meta name="robots" content="noindex,nofollow">

and then change your robots.txt to re-allow them to get to the page.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved