roll-your-own dupe filter? for testing

is there an app out there that measures similarity

[httpwebwitch]

Like most wms, I'd like to stay on the happy side of the dupe filter.

What I'd like is something I can use during development to spider my site and generate a similarity report. I have similar tools for checking broken links and reporting anchor text reputation, but nothing I can use to measure similarity.

Any suggestions?

[httpwebwitch]

I assume from the lack of responses that there is no such thing...

[PizdusInc]

Not sure if it is what you are looking for, but HTMLDiff (HTMLDiff.com) is a program that compares html files for similarities/differences.

[httpwebwitch]

hmm... well not really
that tool will compare files in two folders and show differences, more like version control

I'm looking for something in a web spider that generates a similarity matrix before and after parsing out HTML tags.

In essence, it would replicate the Google dupe filter and alert a WM if any pages don't have enough unique content to be indexed separately.

[PizdusInc]

Hmm, I see what you are saying... I have not seen anything like that around so you will probably have to code something like that yourself. Unless someone else in the forum could help out.

Sorry...

[httpwebwitch]

no don't be sorry, it's all right. I'll make my own, if the effort seems worthwhile.

[Hollywood]

httpwebwitch

If you get it done let me know, I think it would be very worth the time.

;0)

[PizdusInc]

It would definitely be a very useful tool to have... If you end up coding it yourself, are you planning on making it available to the web?

[httpwebwitch]

for a "modest" fee... mmmuaaahahaha ha ha (cough cough)