Link checking software

What do you use

[graywolf]

I was reading another thread and came across the subject of link checking software. Any suggestions on what to use or how to do it so it doesn't eat up your time?

If it's something that you can't post, feel free to send me a sticky mail.

[pendanticist]

That's a bit of a tricky question. :) See many, if not most of the link checking applications are probably on someones ban list.

I've used tons of them, what with the need to constantly verify the URLs of 5,000 +/- sites, and they all have some kind of limitation.

About the only thing that seems to do all that and my "Favs" without having been (or being) banned by those I'm checking is http://www.lithopssoft.com/ HTML Link Validator. Been using both the older version and the newest for nearly two years now and find it works very well.

[rogerd]

Graywolf, do you mean checking the validity of your outbound links (as I think pendanticist interpreted your question), or verifying the existence of an inbound link on a remote site?

[carlwright]

If you're checking validity etc. we use something called LinkBot. Can be a little intensive on the machine running it though.

[graywolf]

verifying the existance of the link

[GaryK]

FWIW, anyone who uses my browscap.ini file and the related code to block what I call "website strippers" both of these link checkers, and more, are considered website strippers and get redirected to an error page with no links on it to follow except an e-mail address for the FTC's spam complaint center. :)

[progex]

Xenu Link Sleuth - Blazingly FAST

[home.snafu.de...]

[rogerd]

Xenu does a great job of link validation, but is disallowed by a fair number of sites. The only way to use it for verification of a remote link would be to use it to spider the remote site and then look at the list of links it found to see if yours is in there. This could be a bit tedious if the other site has a few thousand links.

I think what Graywolf is looking for is a tool that would accept a list of sites or URLs, look for a link to a specified domain, and report back with link confirmation and perhaps link location for each site.

[graywolf]

I think what Graywolf is looking for is a tool that would accept a list of sites or URLs, look for a link to a specified domain, and report back with link confirmation and perhaps link location for each site.

yes that's what I'm looking for.

[zgb999]

GaryK
Thank you for the insight in how you deal with that.

A reason more why to manually check all site before asking for a link.

[pendanticist]

I used Xenu for years and yes, it is good. Does a great little site map for 'ya too.

Alas, she does not check robots.txt and odds are, never will. Many website folks block it just for that reason.

Once the 403's I was being fed reached 2% (out of a total of 5,000 links), I moved away from Xenu. Manually checking even a hundred links every week is a bit tedious.

Oddly enough, those who banned, banned not by UA string, but by IP Number. Ergo, if I can't get there...

Too bad those folks hadn't bothered to see who was using it and why. If they'da just done that....

Now, to the origional question.

If there is such an application currently available, I'd venture a guess it'd be in the 'couple of hundred dollars' (US) range.

[progex]

Ah, ok. Sorry about that.

I did some research and came upon this free software for recip link checking:

LinkCheck

Removed the link since this is commercial software. A crippleware version is freely available.

[edited by: DaveAtIFG at 4:02 pm (utc) on Jan. 2, 2004]