homepage Welcome to WebmasterWorld Guest from 54.227.171.163
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Google indexing robots.txt file
kunwarbs

10+ Year Member



 
Msg#: 3109336 posted 11:19 am on Oct 5, 2006 (gmt 0)

Interesting to see that Google has indexed and cached robots.txt file of reputed websites like nytimes, BBC and Google itself...

[google.com...]

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3109336 posted 11:38 am on Oct 5, 2006 (gmt 0)

ya, mentioned alot in the 4 years they've been doing it.

NedProf

5+ Year Member



 
Msg#: 3109336 posted 12:02 pm on Oct 5, 2006 (gmt 0)

Is that because of the text/html mime-type in stead of the text/plain that it should be?

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3109336 posted 7:47 pm on Oct 5, 2006 (gmt 0)

It's because someone somewhere links to that file, so they treat it as content as well as its true purpose.

Jordo needs a drink

5+ Year Member



 
Msg#: 3109336 posted 2:08 am on Oct 6, 2006 (gmt 0)

It's because someone somewhere links to that file, so they treat it as content as well as its true purpose.

The best example is in the search results you posted. #1 is Wikipedia expaining robots.txt. #2 is the White House robots.txt itself.

Look again at at the Wiki article and you'll see they link to the White House robots.txt

etgsgroup

5+ Year Member



 
Msg#: 3109336 posted 3:23 am on Oct 6, 2006 (gmt 0)

Why Google database show robots.txt file?

GaryK

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3109336 posted 3:38 am on Oct 6, 2006 (gmt 0)

Why Google database show robots.txt file?

That sure seems like an obvious question to me too. It seems like it would be simple enough for Google to implement. Is anyone really interested in seeing the contents of a robots.txt file in their SE results?

Tastatura

5+ Year Member



 
Msg#: 3109336 posted 4:00 am on Oct 6, 2006 (gmt 0)

Number 4 is BT's robots.txt blog :)
number 5 is google's own robots txt file


Webmasterworld: Robots.txt
Brett Tabke experiments with writing a weblog in a text file usually read only by robots. Trenchant commentary on the world of search engine marketing.
www.webmasterworld.com/robots.txt - 2k - Cached - Similar pages

google's robots txt - [ Translate this page ]
User-agent: * Allow: /searchhistory/ Disallow: /news?output=xhtml& Allow: /news?output=xhtml Disallow: /search Disallow: /groups Disallow: /images Disallow: ...
www.google.com/robots.txt - 3k - Cached - Similar pages


smells so good

5+ Year Member



 
Msg#: 3109336 posted 4:23 am on Oct 6, 2006 (gmt 0)

It's one of the few ways that Brett will have his blog found.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved