ReSiever - 3:14 pm on Mar 15, 2005 (gmt 0)

Hi everybody..

I know their have been a few topics about this one, but i can't figure out what the Mozilla/5.0 Google spider exactly does..

Everything that gets spidered is not getting indexed, so, it's not a regular spider. It's meant specifically for something else. Cloaking detection could be one, but i can't see why they would build a whole new spider just for the detection of cloaked pages. Next to that, even if SEO's cloak pages, they would know by now their is a spider going around detecting it, so the cloaking would be done differently, undetectable.

Maybe the spider is used for analysing content of pages? This could be for future use (LSA?), to detect duplicate content or maybe to see if pages in a linkstructure contain mostly the same content..

I wonder what your thoughts are, your findings etc, cause this spider seems smarter to me then we might think it is...