Page is a not externally linkable
- Google
-- Google News Archive
---- Everflux - October 2002 mid-month Google spidering and minor updates
crash - 4:22 pm on Oct 3, 2002 (gmt 0)
Google honors the robots.txt file. disallow the pages you don't want crawled.
Thread source:: http://www.webmasterworld.com/google_archive/5914.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com