Forum Moderators: goodroi
It’s quite strange, I’ve got this as my robots.txt
User-agent: *
Disallow: /go/
When I do allinurl:www.example.co.uk at google it lists a few urls like www.example.co.uk/go/124 there are over a hundred of similar urls but only lists a few. Looking at my logs the googlebot has obeyed robots.txt as I can’t see any visits by google to those pages.
How have they got in the index, Have I messed up the robots file?
Cheers
There was a major hoo-ha on these forums when they started doing it...
G started "listing" pages they weren't allowed to visit at some point in the recent past. No descriptions, just the url.
Technically speaking they don't list (or have) pages in index, but just URLs pointing to those pages. These URLs can be found on pages that point to your website and robots.txt does not control that process, so from robots.txt compliance point of view all is perfectly legit.
If you don't want search engine to have pages or URLs in their index then I think you might have to use META command with nofollow, noindex, but of course if you have robots.txt preventing search engine from getting actual pages to find META command then they would not have a clue and assume that its okay to do what they do.
