Google ignoring robots.txt

No real surprise, but a real pain

[voice220]

On several pages on of our sites, there are links that have everchanging URLs. They all go via a specific link directory (eg. /go/places). Our robots.txt says to ignore all /go/ and beyond places. Google on the other hand ignores that completely it seems, and indexes those ephemeral pages. Any ideas?

[ukgimp]

is your robots.txt correct in its syntax. You may be at a fault, not google.

[voice220]

I thought about that and had another look. This is what it looks like, and there's really not that much to do wrong in those two lines :?

User-agent: *
Disallow: /go/

[ukgimp]

[searchengineworld.com...]

[voice220]

Thanks, but it validates just fine.

[spud01]

No errors detected! This Robots.txt validates to the robots exclusion standard!

So is there any truth to google not obeying the robots.txt file?

This could also explain why our site has lost any relevance on google with no top 100 position for all the relevant keywords we optimized for.

The site is optimized for 2 different resolution (800x600 & 1024x768) and hence there is duplicate content, but the duped pages/folders are disallowed in the robots.txt file.

[voice220]

It would appear so, because our disallowed pages are happily indexed.

[DaveAtIFG]

There's a similar discussion at [webmasterworld.com...]

Welcome to WebmasterWorld voice220! :)

[voice220]

Thanks DaveAtIFG, the above link explained a lot, and may just have solved my dilemna. :)