Forum Moderators: goodroi

Message Too Old, No Replies

Google - robots.txt - indexing

         

Brett_Tabke

10:01 am on Aug 21, 2001 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Oops, hmm - Google indexed our robots.txt [google.com]. That's the first time I've ever seen that.

starec

10:56 am on Aug 21, 2001 (gmt 0)

10+ Year Member



Well, you put a link to it some time ago in one of the forums, didn't you?

(www.webmasterworld.com/forum27/239.htm)

With the pagerank this site has anything that is linked from here gets indexed easily :)

Key_Master

11:17 am on Aug 21, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Brett_Tabke, did you add the HTML or did Google?

highman

11:52 am on Aug 21, 2001 (gmt 0)

10+ Year Member



'Please, no more that 1 hit per minute and use off peak hours.' .... lol, must remember to add it to our robots.txt along with 'please rank us very highly'

Brett_Tabke

3:01 pm on Jan 25, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Nah, it's a pure text file KM.

gethan

3:37 pm on Jan 25, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



a few links [google.com].

Your robots.txt has got a PR of 4 ;)

Brett_Tabke

9:24 am on Jun 24, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



It still perplexes me that a robots.txt would be indexed. hmmm

Anyone else got a robots.txt index?

SmallTime

9:38 am on Jun 24, 2002 (gmt 0)

10+ Year Member



Google, CNN, W3.org, your in good company

Woz

9:40 am on Jun 24, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



funny, first time I clicked on the link in the first post it came up showing zero responses. Then, on second attempt, it came up. Perhaps still quivering after the aborted update.

Onya
Woz