Page is a not externally linkable
- Google
-- Google News Archive
---- Google.com now has no robots.txt?


lazerzubb - 3:20 pm on Sep 1, 2003 (gmt 0)


I think it only means an increase in the index.

Google have always been very well on spidering serp's on google.com itself, maybe that's the "the" reason.

Fun thing to do is to enter portals and search engines in google and do a SITE: on them, and see how many of the results they've indexed, then check out the robots.txt on those sites ;)


Thread source:: http://www.webmasterworld.com/google_archive/16587.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com