homepage Welcome to WebmasterWorld Guest from 23.22.2.150
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Google - robots.txt - indexing
Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34 posted 10:01 am on Aug 21, 2001 (gmt 0)

Oops, hmm - Google indexed our robots.txt [google.com]. That's the first time I've ever seen that.

 

starec

10+ Year Member



 
Msg#: 34 posted 10:56 am on Aug 21, 2001 (gmt 0)

Well, you put a link to it some time ago in one of the forums, didn't you?

(www.webmasterworld.com/forum27/239.htm)

With the pagerank this site has anything that is linked from here gets indexed easily :)

Key_Master

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 34 posted 11:17 am on Aug 21, 2001 (gmt 0)

Brett_Tabke, did you add the HTML or did Google?

highman

10+ Year Member



 
Msg#: 34 posted 11:52 am on Aug 21, 2001 (gmt 0)

'Please, no more that 1 hit per minute and use off peak hours.' .... lol, must remember to add it to our robots.txt along with 'please rank us very highly'

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34 posted 3:01 pm on Jan 25, 2002 (gmt 0)

Nah, it's a pure text file KM.

gethan

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 34 posted 3:37 pm on Jan 25, 2002 (gmt 0)

a few links [google.com].

Your robots.txt has got a PR of 4 ;)

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34 posted 9:24 am on Jun 24, 2002 (gmt 0)

It still perplexes me that a robots.txt would be indexed. hmmm

Anyone else got a robots.txt index?

SmallTime

10+ Year Member



 
Msg#: 34 posted 9:38 am on Jun 24, 2002 (gmt 0)

Google, CNN, W3.org, your in good company

Woz

WebmasterWorld Senior Member woz us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 34 posted 9:40 am on Jun 24, 2002 (gmt 0)

funny, first time I clicked on the link in the first post it came up showing zero responses. Then, on second attempt, it came up. Perhaps still quivering after the aborted update.

Onya
Woz

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved