Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Is google ignoring ROBOTS.TXT?!

13,000 pages indexed, ALL blocked by ROBOT.TXT

         

Tearabite

2:48 am on Aug 8, 2006 (gmt 0)

10+ Year Member



I just did a SITE:mysite.com and to my surprise found over 13,000 pages listed! (a big surprise considering i only have about 200 pages on my site)
On closer inspection, i see that many of these pages are in my forum area - all of which are are blocked by my robots.txt.

I used the robots.txt validator in sitemaps, and it confirms these pages are blocked, and yet they are still being indexed... to add insult to injury, now many of my other pages are going supplimental..

is this some known issue, or am I doing something wrong?!

Tearabite

4:10 am on Aug 8, 2006 (gmt 0)

10+ Year Member



OK.. someone please explain this to me..

after my initial panic at seeing SUPPLIMENTAL all over my site, as well as thousands of pages that were supposed to be blocked by ROBOTS.TXT, i remembered something..

Last week while searching for some.. ehm.."pictures", i changed my G search preferences to "all languages" (dont want to miss out any any good Italian or Greek pics!).. So just now I changed my preference back to English only, and now when i do a SITE:mysite, everything looks fine... i switch back to "all languages" in preferences, and the thousands of pages that are supposed to be blocked, as well as the "supplemental" are back.. switch back to English only, and everything is fine once again..

Is this somehow by design? does this happen to anyone else?

daveVk

4:30 am on Aug 8, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes ENGLISH means no sups. for me too. Have url only entries for pages excluded by robots.txt but indexed prior to robots.txt in place.

smokeybarnable

1:12 pm on Aug 8, 2006 (gmt 0)

10+ Year Member



I tried setting to english only results and all my supplemental pages disappeared. why?