jimbeetle - 5:08 pm on Jul 17, 2010 (gmt 0)
He, he, he. This is what you're thinking about at 2:05 in the AM!?!
As part of that dark matter I still firmly believe that the supplemental index still exists and that a URL that "falls out" of the main index has simply "gone supplemental" as we used to say. And following on lammert's comment I'll even posit a NOINDEX index. And a 404 index. And then a...whatever. Yeah, plenty of stuff we can't see.
Bottom line though, I think Google would have to consider all of the URLs it knows about that are capable of passing link juice in its iterations simply because if it did not it would be creating innumerable dead ends where the juice couldn't flow back into the system. I ain't no math guy, but after umpty-ump iterations wouldn't the PR of the entire web then be effectively reduced to zero?
Just yesterday I reread Saul Hansell's 2007 NYT article Google Keeps Tweaking Its Search Engine [nytimes.com]. I'm still quite astounded that search professionals and academics were astounded at what Google was doing two years ago. Guess my hope is actually not understanding G, simply surviving in spite of my not understanding it.