mack - 10:33 pm on Jan 8, 2007 (gmt 0)

The bot generally downloads the entire page to a server at the SE. What the SE generally ends up with is a local copy of the indexible web.

It is locally on the se server clusters that the real number crunching begins. Links, anchor, content analysis etc are all taken into account at this stage.

One think I think we can also take into account is the search engine spider update schedule. Most search engines use the Last modified header to determines if a page has updated. It makes little or no sense to re-index pages that have not changed.

Mack.