Page is a not externally linkable
pageoneresults - 3:41 pm on Jan 8, 2007 (gmt 0)
Once the multi-tasking bot requests a URI, the first thing it does is request the server headers, correct? From there it requests the <head></head> of the document, correct? And then from there it traverses the html/xhtml stripping out all markup and ending up with one big chunk of text, correct?
So, we have a multi-tasking bot. Actually, we have a network of bots each indexing specific content. Okay, that answers one question.