Welcome to WebmasterWorld Guest from 54.226.2.31

Message Too Old, No Replies

Google indexes ROBOTS.TXT

     

kaled

10:11 am on Sep 8, 2005 (gmt 0)

WebmasterWorld Senior Member kaled is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I know posting specific searches is frowned upon, but try searching for <snip> from google.co.uk

The second result is their robots.txt file - well, it made me laugh.

Kaled.

[edited by: Brett_Tabke at 1:36 pm (utc) on Sep. 9, 2005]
[edit reason] lets not do specifics... [/edit]

Brett_Tabke

1:37 pm on Sep 9, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Ya, alot of sites get their robots.txt indexed if they get pointed at.

Look at #4:
[google.com...]

stinkfoot

1:46 pm on Sep 9, 2005 (gmt 0)

10+ Year Member



Having asked about this before the real question is why ... a txt file should surely be seen and not heard. It is almost as bad a putting an xml file in the serps.

txt = no formatting
xml = so much foramtting you cant read it

Both are silly results imo

Lord Majestic

1:50 pm on Sep 9, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Both are silly results imo

They don't have to have high rank, and if there is no other good matches then better have text file or even XML than no matches at all?

Brett_Tabke

2:03 pm on Sep 9, 2005 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



if it is digital, online, and accessible - google is going to index it no matter what.

siteseo

2:05 pm on Sep 9, 2005 (gmt 0)

10+ Year Member



I've heard G-men state that they actually consider .doc files to be highly relevant because they generally consist of nothing but text. I imagine the same could be true for .txt files.

vincevincevince

6:57 pm on Sep 12, 2005 (gmt 0)

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member



There are some very informative documents in .txt, particularly product and software manuals / FAQ files / Release/Change notes, etc.

victor

7:09 pm on Sep 12, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Reserving robots.txt as a file only for web bots is a convention. Its usage is neither compulsary nor universal.

If a site has a link to its robots.txt, then it is directing non-bot visitors to it. Makes it fair game for search engines to index.

caveman

7:18 pm on Sep 12, 2005 (gmt 0)

WebmasterWorld Senior Member caveman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



> online, and accessible

Not everyone understands what this implies. A lot of things get indexed that surprise people.

We're much more careful now than we used to be, having got a few nasty surprises ourselves.

Kirby

7:53 pm on Sep 12, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>Not everyone understands what this implies. A lot of things get indexed that surprise people.

use a program like teleport and it may be surprising to a lot of webmasters what it finds that they didnt think was accesible. I assume gbot is far more effecient at finding things.

jgbmarc

7:55 pm on Sep 12, 2005 (gmt 0)

10+ Year Member



plug brett's search into google.. i wonder why the whitehouse doesnt want ALL that content crawled?

henweb

2:05 pm on Sep 13, 2005 (gmt 0)

10+ Year Member



Here's a question I thought of while doing a google search on "robots.txt" file - why does the whitehouse site (which comes up about third or fourth) have a /text and /iraq extension on every url they're asking google to ignore? What's THAT about?

Henry

 

Featured Threads

Hot Threads This Week

Hot Threads This Month