homepage Welcome to WebmasterWorld Guest from 54.196.63.93
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How do I not allow google
pizzaguy

5+ Year Member



 
Msg#: 3468385 posted 9:22 pm on Oct 3, 2007 (gmt 0)

Hi folks, a new section is coming for my web site. It is not finished yet and I would like some of my visitors to access this page so that I can figure out what else I have to do on the web site according to this info.

The only thing is I want this page to be accessible by my visitors but I do not want Google or other robots to index these pages. How do I allow this? I read that no index tag works but google will ignore it. Is that true? I would appreciate if somebody can advice me about this.

Thanks in advance

 

jimbeetle

WebmasterWorld Senior Member jimbeetle us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3468385 posted 10:31 pm on Oct 3, 2007 (gmt 0)

What Gbot sometimes appears to "ignore" is a disallow directive in robots.txt. If the 'bot finds enough links to a disallowed page it thinks the page is somewhat important and will include it in the index as a "URL only" listing, basically naked with not title or description.

To be sure a page isn't included the meta noindex is the best bet. Just be sure not to also disallow the section in robots.txt as the 'bot has to be able to read the pages in order to obey the noindex.

I'd probably also slap a nofollow on it:

<META name="ROBOTS" content="NOINDEX,NOFOLLOW">

I used the generic ROBOTS instead of the specific GOOGLEBOT because, well, you know, there are other search engines out there ;-).

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved