Page is a not externally linkable
TallTroll - 2:18 pm on Mar 16, 2004 (gmt 0)
Stripping the retrieved documents for links and attempting to "understand" the retrieved documents are totally separate operations in the search engines processing sequence. For those who want to waste an hour on a fascinating examination of search theory, go to G, and perform this search [google.com] Google have over 300k .js files "indexed". They just don't UNDERSTAND any of them. Try swapping out other filetypes. You'll be amazed what you turn up. If you watch REALLY carefully, you'll find an unintentional translation to Googles internal usage operator, ext:{insert file extension here}
Google (and I assume other engines) will attempt to request ANY URI of the form "http://www.example.com" found in any document they crawl.