Page is a not externally linkable
- Google
-- Google News Archive
---- Google.com now has no robots.txt?


Key_Master - 9:22 pm on Sep 1, 2003 (gmt 0)


jesserud,

What you fail to understand is a robot can follow any www.google.com link it finds (or on any subdomain at Google for that matter) if it can't find a robots.txt at google.com. Robots.txt files are not supposed to redirect. Even GoogleGuy would agree with that.

But I think they're working on fixing the problem now.


Thread source:: http://www.webmasterworld.com/google_archive/16587.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com