Google indexes ROBOTS.TXT

[kaled]

I know posting specific searches is frowned upon, but try searching for <snip> from google.co.uk

The second result is their robots.txt file - well, it made me laugh.

Kaled.

[edited by: Brett_Tabke at 1:36 pm (utc) on Sep. 9, 2005]
[edit reason] lets not do specifics... [/edit]

[Brett_Tabke]

Ya, alot of sites get their robots.txt indexed if they get pointed at.

Look at #4:
[google.com...]

[stinkfoot]

Having asked about this before the real question is why ... a txt file should surely be seen and not heard. It is almost as bad a putting an xml file in the serps.

txt = no formatting
xml = so much foramtting you cant read it

Both are silly results imo

[Lord Majestic]

Both are silly results imo

They don't have to have high rank, and if there is no other good matches then better have text file or even XML than no matches at all?

[Brett_Tabke]

if it is digital, online, and accessible - google is going to index it no matter what.

[siteseo]

I've heard G-men state that they actually consider .doc files to be highly relevant because they generally consist of nothing but text. I imagine the same could be true for .txt files.

[vincevincevince]

There are some very informative documents in .txt, particularly product and software manuals / FAQ files / Release/Change notes, etc.

[victor]

Reserving robots.txt as a file only for web bots is a convention. Its usage is neither compulsary nor universal.

If a site has a link to its robots.txt, then it is directing non-bot visitors to it. Makes it fair game for search engines to index.

[caveman]

> online, and accessible

Not everyone understands what this implies. A lot of things get indexed that surprise people.

We're much more careful now than we used to be, having got a few nasty surprises ourselves.

[Kirby]

>Not everyone understands what this implies. A lot of things get indexed that surprise people.

use a program like teleport and it may be surprising to a lot of webmasters what it finds that they didnt think was accesible. I assume gbot is far more effecient at finding things.

[jgbmarc]

plug brett's search into google.. i wonder why the whitehouse doesnt want ALL that content crawled?

[henweb]

Here's a question I thought of while doing a google search on "robots.txt" file - why does the whitehouse site (which comes up about third or fourth) have a /text and /iraq extension on every url they're asking google to ignore? What's THAT about?

Henry