TheWhippinpost - 9:22 pm on Feb 10, 2007 (gmt 0)

Once again, the comprehensive nature of the technology over a simplified model such as LSI is obvious.

I too have expected something along the lines of this phrase theory, though I see it as an extension to an LSI-type algo.

Whereas LSI essentially talks about synomyns of words, this almost lends itself to "synomyns of phrases"

If you were to compare the pages of a product tutorial, a product review, and a typical legit ecommerce product page (we'll assume it's a tech kind of product here), you would most likely see a far higher density of technical language being used in the merchants page, than either of the others.

What's more, the proximity, ie... the "distance", between each of those technical words, are most likely to be far closer together on the merchants page too (think product specification lists etc...).

Tutorial pages will have a higher incidence of "how" and "why" types of words and phrases.

Reviews will have more qualitative and experiential types of words ('... I found this to be robust and durable and was pleasantly surprised...').

Sales pages similarly have their own (obvious) characteristics.

Mass-generated spammy pages that rely on scraping and mashing-up content to avoid dupe filters whilst seeding in the all-important link-text (with "buy" words) etc... should, in theory, stand-out amongst the above, since the spam will likely draw from a mixture of all the above, in the wrong proportions.

Therefore the associated phrases need not be on the same page, but in the cluster of pages and the overall density and frequency valued over the whole cluster.

Most definitely.