Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google ignoring robots.txt and noindex metatags

         

Mokita

3:16 pm on Oct 1, 2006 (gmt 0)

10+ Year Member



Recently, Google has begun listing URLs of ours in the SERPS which are both disallowed in robot.txt and which have <meta name="robots" content="noindex"> in the headers. These URLs have been consistently disallowed for 18 months or more - they haven't just been added to robots.txt recently.

Yet, Webmaster Tools correctly shows a number of them as "URLs restricted by robots.txt" on dates within the last couple of weeks. Also, when I "Test URLs against this robots.txt file", it correctly reports all of them as being "Blocked" to Googlebot.

Admittedly Google is only listing the URL with no snippet, but they do appear when I do a general search for terms contained in the URL and then click on "repeat the search with the omitted results included."

I realise this is not exactly disastrous, but isn't it totally contrary to the intention of robots.txt and the robots meta tag?

I really don't want any of these URLs appearing in the serps, but have no idea how to stop Google from listing them. Does anyone have any suggestions or explanation as to why Google is doing this?

g1smd

10:14 am on Oct 2, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




This is an obvious bug in Google's Supplemental Results handling.

It has happened before. The last time was only a few months ago.
It was first noticed in June, and appears to have been fixed by August.

[webmasterworld.com...] - Very relevant.

[webmasterworld.com...] - Not so relevant.

Is your case the same as that previously described here?