4specs - 5:39 pm on Feb 23, 2007 (gmt 0)

I maintain a library service directory with over 13,000 links. My goal is under 1% problem links. I do it this way:

1. Run Xenu at the beginning of the month
2. Run a Perl script on an outside server at the middle of themonth that downloads the home page of each site. It logs any html refreshes and the page title. Every 6 months I have my father in law look at questionable or blank titles.
3. Every 18 months or so my father in law looks at each site
4. We have a phone call to each company every 24-36 months. Again my father in law. A great way to write him a check he feels good about.

Xenu shows up sites not responding and 301 and 302s.

The Perl script shows up meta-refreshes to new domains and NetSol expired domains and other probable problems indicated by the page title.

What this misses:

1. Dumb web designers doing a javascript refresh to a new domain or just a link on the page - we find these with a manual check.
2. Companies no longer in existance and the isp or web designer has their website still up.