Hi folks, a new section is coming for my web site. It is not finished yet and I would like some of my visitors to access this page so that I can figure out what else I have to do on the web site according to this info.
The only thing is I want this page to be accessible by my visitors but I do not want Google or other robots to index these pages. How do I allow this? I read that no index tag works but google will ignore it. Is that true? I would appreciate if somebody can advice me about this.
What Gbot sometimes appears to "ignore" is a disallow directive in robots.txt. If the 'bot finds enough links to a disallowed page it thinks the page is somewhat important and will include it in the index as a "URL only" listing, basically naked with not title or description.
To be sure a page isn't included the meta noindex is the best bet. Just be sure not to also disallow the section in robots.txt as the 'bot has to be able to read the pages in order to obey the noindex.
I'd probably also slap a nofollow on it:
<META name="ROBOTS" content="NOINDEX,NOFOLLOW">
I used the generic ROBOTS instead of the specific GOOGLEBOT because, well, you know, there are other search engines out there ;-).