Welcome to WebmasterWorld Guest from 54.167.85.221

Forum Moderators: goodroi

Message Too Old, No Replies

Google ignoring robots.txt

No real surprise, but a real pain

     

voice220

8:53 am on Jan 19, 2004 (gmt 0)

10+ Year Member



On several pages on of our sites, there are links that have everchanging URLs. They all go via a specific link directory (eg. /go/places). Our robots.txt says to ignore all /go/ and beyond places. Google on the other hand ignores that completely it seems, and indexes those ephemeral pages. Any ideas?

ukgimp

8:56 am on Jan 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



is your robots.txt correct in its syntax. You may be at a fault, not google.

voice220

9:31 am on Jan 19, 2004 (gmt 0)

10+ Year Member



I thought about that and had another look. This is what it looks like, and there's really not that much to do wrong in those two lines :?

User-agent: *
Disallow: /go/

ukgimp

9:36 am on Jan 19, 2004 (gmt 0)

voice220

9:56 am on Jan 19, 2004 (gmt 0)

10+ Year Member



Thanks, but it validates just fine.

spud01

10:12 am on Jan 19, 2004 (gmt 0)

10+ Year Member



No errors detected! This Robots.txt validates to the robots exclusion standard!

So is there any truth to google not obeying the robots.txt file?

This could also explain why our site has lost any relevance on google with no top 100 position for all the relevant keywords we optimized for.

The site is optimized for 2 different resolution (800x600 & 1024x768) and hence there is duplicate content, but the duped pages/folders are disallowed in the robots.txt file.

voice220

10:31 am on Jan 19, 2004 (gmt 0)

10+ Year Member



It would appear so, because our disallowed pages are happily indexed.

DaveAtIFG

3:37 pm on Jan 19, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



There's a similar discussion at [webmasterworld.com...]

Welcome to WebmasterWorld voice220! :)

voice220

4:12 pm on Jan 19, 2004 (gmt 0)

10+ Year Member



Thanks DaveAtIFG, the above link explained a lot, and may just have solved my dilemna. :)
 

Featured Threads

Hot Threads This Week

Hot Threads This Month