Google webmaster tools reports that the top 3 most common keywords found in my site are 'Disallow', 'User' and 'Agent', which clearly only exist in my 'robots.txt' file. Why is Google listing these words and how do I prevent it from doing so please?
Part of the answer is simple: As noted in earlier threads, g### does index robots.txt itself.
It is appropriate to allow everyone access to robots.txt. Again, see assorted other threads. But you'd be better with a FilesMatch if you want to group them. I think Apache itself says so. I group mine by extension:
<FilesMatch "(forbidden|goaway|missing)\.html"> Order Allow,Deny Allow from all </FilesMatch>
<Files "robots.txt"> Order Allow,Deny Allow from all </Files>
<Files "favicon.ico"> Order Allow,Deny Allow from all </Files>
By "part of" I meant the discussion about material from robots.txt showing up in searches. Google also indexes sitemaps and-- if you're careless enough to leave them lying around-- raw logs. They would undoubtedly index your htaccess if only they could get to it.
But if the single most common words are "Disallow" "User" and "Agent" it suggests that they haven't got around to counting the rest of the keywords yet. Feed in some random exact-text phrases and make sure they come up in search results. Then you'll know that your other pages are indexed.
Keywords seem to be processed entirely separately from indexing-in-general. And I've got a hunch they don't generate the list at all until you sign up with gwt. In my case I'd just gotten used to a list absurdly packed with words like "it's" and, yes, "word"... and then suddenly a whole slew of names crops up. They belong to a rarely-visited page that happens to be fatter (in html) than anything else on the site. So as soon as they threw it into the Keywords mix, everything changed.
For a while, one of my most common keywords was "thumbnail". I finally forced myself to sit down and make proper alts for all my, ahem, thumbnails ;)