homepage Welcome to WebmasterWorld Guest from 54.211.73.232
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Google indexes ROBOTS.TXT
kaled

WebmasterWorld Senior Member kaled us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 31129 posted 10:11 am on Sep 8, 2005 (gmt 0)

I know posting specific searches is frowned upon, but try searching for <snip> from google.co.uk

The second result is their robots.txt file - well, it made me laugh.

Kaled.

[edited by: Brett_Tabke at 1:36 pm (utc) on Sep. 9, 2005]
[edit reason] lets not do specifics... [/edit]

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 31129 posted 1:37 pm on Sep 9, 2005 (gmt 0)

Ya, alot of sites get their robots.txt indexed if they get pointed at.

Look at #4:
[google.com...]

stinkfoot

10+ Year Member



 
Msg#: 31129 posted 1:46 pm on Sep 9, 2005 (gmt 0)

Having asked about this before the real question is why ... a txt file should surely be seen and not heard. It is almost as bad a putting an xml file in the serps.

txt = no formatting
xml = so much foramtting you cant read it

Both are silly results imo

Lord Majestic

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 31129 posted 1:50 pm on Sep 9, 2005 (gmt 0)

Both are silly results imo

They don't have to have high rank, and if there is no other good matches then better have text file or even XML than no matches at all?

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 31129 posted 2:03 pm on Sep 9, 2005 (gmt 0)

if it is digital, online, and accessible - google is going to index it no matter what.

siteseo

10+ Year Member



 
Msg#: 31129 posted 2:05 pm on Sep 9, 2005 (gmt 0)

I've heard G-men state that they actually consider .doc files to be highly relevant because they generally consist of nothing but text. I imagine the same could be true for .txt files.

vincevincevince

WebmasterWorld Senior Member vincevincevince us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 31129 posted 6:57 pm on Sep 12, 2005 (gmt 0)

There are some very informative documents in .txt, particularly product and software manuals / FAQ files / Release/Change notes, etc.

victor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 31129 posted 7:09 pm on Sep 12, 2005 (gmt 0)

Reserving robots.txt as a file only for web bots is a convention. Its usage is neither compulsary nor universal.

If a site has a link to its robots.txt, then it is directing non-bot visitors to it. Makes it fair game for search engines to index.

caveman

WebmasterWorld Senior Member caveman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 31129 posted 7:18 pm on Sep 12, 2005 (gmt 0)

> online, and accessible

Not everyone understands what this implies. A lot of things get indexed that surprise people.

We're much more careful now than we used to be, having got a few nasty surprises ourselves.

Kirby

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 31129 posted 7:53 pm on Sep 12, 2005 (gmt 0)

>Not everyone understands what this implies. A lot of things get indexed that surprise people.

use a program like teleport and it may be surprising to a lot of webmasters what it finds that they didnt think was accesible. I assume gbot is far more effecient at finding things.

jgbmarc

5+ Year Member



 
Msg#: 31129 posted 7:55 pm on Sep 12, 2005 (gmt 0)

plug brett's search into google.. i wonder why the whitehouse doesnt want ALL that content crawled?

henweb

5+ Year Member



 
Msg#: 31129 posted 2:05 pm on Sep 13, 2005 (gmt 0)

Here's a question I thought of while doing a google search on "robots.txt" file - why does the whitehouse site (which comes up about third or fourth) have a /text and /iraq extension on every url they're asking google to ignore? What's THAT about?

Henry

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved