Google and noindex, follow

oddity or not logical

[Blade]

A friend of mine has encounter a somewhat puzzling oddity with google and other spiders and the robots meta tags.

He has 2 pages that don’t want indexing because the content is not significant.

What he has is the following linking through to the end page 3:

PAGE 1: <meta name="robots" content="noindex,follow" />
PAGE 2: <meta name="robots" content="noindex,follow" />
PAGE 3: All the good content is here for spidering and indexing.

BUT, google seems not to go past the second page thus nothing is indexed!

I *think* this ordering is logically correct, any opinions on this?

[fathom]

PAGE 1: <meta name="robots" content="noindex,follow" />

On entering your site each time Google will check your robots.txt so any pages that you don't wish indexed should be place here.

Assuming your site is XML Google will understand your mata tag - however, if HTML it should be <meta name="robots" content="noindex,follow"> without a space and forward-slash to close the tag.

[Blade]

Thanks,

but if no robots.txt or a blank robots.txt is present, then the spider should then look for the robots meta tag in the pages themselves. As such the above should instruct where and what not to spider.

I'm wondering if having 2 noindex,follow is a bad thing as google wont follow two in succession like this.

[fathom]

Personally Blade - I would start with all pages being indexable unless there is a privacy issue involved.

If just a low level "non-content page" it just won't rank high for any qualified search terms - and you can live with that.

Site breadth is important since these pages normally link back to pages that you wish to rank on... and therefore the anchors back add relevancy to good content.

[Blade]

Yep, this is more important than the pages not showing up - its only low level content after all. Perhaps relying on engines following strict robot meta rules is not such a good idea as I know of some smaller engines that have patchy support for these.

Thanks for the tip.

[Oaf357]

The trailing slash in his tag is XHTML. Not quite sure how Google is handling it but at this time it doesn't seem that Google really links XHTML at all.

Hopefully, the mighty Google will catch up.