incrediBILL - 4:40 pm on Feb 23, 2007 (gmt 0)

Ah yes, a couple of more tips...

CACHE THE PAGES - If you CACHE all of the pages when you make a full pass validating your directory you can then test various new finger prints rapidly. Using the cached pages on your local hard disk instead of going back and loading each page from each individual web servers can help to rapidly test new code. However, I wouldn't rely on cached pages for more than 24 hours for this purpose.

RUN A SEARCH SITE? What I'm describing isn't just useful for a directory as search engines do similar things to detect broken sites in their index. A conversation I had with an engineer from Live Search confirmed that they too do all the soft 404 checking as well. I was looking for some additional pointers on domain park detection and the guy suddenly got really tight lipped as it appears they see that info as highly confidential, as they don't want the domain parks to figure out how they're being identified.

[edited by: incrediBILL at 4:42 pm (utc) on Feb. 23, 2007]