homepage Welcome to WebmasterWorld Guest from 184.72.82.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Google - robots.txt - indexing
Brett_Tabke




msg:1529223
 10:01 am on Aug 21, 2001 (gmt 0)

Oops, hmm - Google indexed our robots.txt [google.com]. That's the first time I've ever seen that.

 

starec




msg:1529224
 10:56 am on Aug 21, 2001 (gmt 0)

Well, you put a link to it some time ago in one of the forums, didn't you?

(www.webmasterworld.com/forum27/239.htm)

With the pagerank this site has anything that is linked from here gets indexed easily :)

Key_Master




msg:1529225
 11:17 am on Aug 21, 2001 (gmt 0)

Brett_Tabke, did you add the HTML or did Google?

highman




msg:1529226
 11:52 am on Aug 21, 2001 (gmt 0)

'Please, no more that 1 hit per minute and use off peak hours.' .... lol, must remember to add it to our robots.txt along with 'please rank us very highly'

Brett_Tabke




msg:1529227
 3:01 pm on Jan 25, 2002 (gmt 0)

Nah, it's a pure text file KM.

gethan




msg:1529228
 3:37 pm on Jan 25, 2002 (gmt 0)

a few links [google.com].

Your robots.txt has got a PR of 4 ;)

Brett_Tabke




msg:1529229
 9:24 am on Jun 24, 2002 (gmt 0)

It still perplexes me that a robots.txt would be indexed. hmmm

Anyone else got a robots.txt index?

SmallTime




msg:1529230
 9:38 am on Jun 24, 2002 (gmt 0)

Google, CNN, W3.org, your in good company

Woz




msg:1529231
 9:40 am on Jun 24, 2002 (gmt 0)

funny, first time I clicked on the link in the first post it came up showing zero responses. Then, on second attempt, it came up. Perhaps still quivering after the aborted update.

Onya
Woz

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved