Page is a not externally linkable
- Google
-- Google News Archive
---- Google.com now has no robots.txt?


Key_Master - 8:01 pm on Sep 1, 2003 (gmt 0)


google.com (no www) doesn't need a robots.txt because every page there redirects to the same page on www.google.com

Google doesn't use invisible redirects so while your browser may follow a 301 or 302 redirect, a robot will not.

As it currently stands, any bot looking for a robots.txt on google.com (without the www) will not find one and will have every right under the standards set forth under the robots.txt exclusion protocol to fully spider any portion of their website.


Thread source:: http://www.webmasterworld.com/google_archive/16587.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com