Link checkers most frequent spiders in my sites

Do you ban them?

[silverbytes]

I sadly noticed that those are becoming more popular in my site as visitors than google and yahoo:

Xenu Link Sleuth
linkwalker
LinksManager

If I'm right those are link checkers.
Do you actually ban those? I understant that linkpartners using it will not see your links to them if you do. In the other hand I don't like the idea of being visited so often for those spiders.

Any comments?

[kevinpate]

They don't come around my corner of the prairie no more. I suspect grazing on 403's left a bad taste on their mouths.

[GaryK]

From my notes:
LinksManager - well behaved, not banned.
linkwalker - well behaved, not banned.
Xenu - This is, or at least can be a very disrespectful and harmful link checker. Banned.

[silverbytes]

Do you know what spider linkpartners com uses? I have some partners there and maybe banning theirs bots is not good idea...

[larryhatch]

As a rule, I would not ban link checkers. Suppose they can't get in?
They MIGHT interpret this as a dead link. You could throw away good incoming links.
People check their links for a reason. They pitch the dead / inactive ones. -Larry

[GaryK]

In general I agree with you Larry. But not all link checkers are created equally.

Take Xenu as an example.

It's a legitimate tool for checking for broken links on your own site before it goes live.

But it can also be used to harvest links from other sites and it usually does not read, and hence it does not respect robots.txt.

In general a link checker doesn't need to read robots.txt, but if it's going to essentially spider someone else's site it should read and respect that file.

[DanA]

Xenu as a link checker only uses HEAD requests.
You don't have to ban it in that case.
If its (bad) user decides to make a map of your site, it will use GET, and parse the HTML but it will not make a local copy. In that case, give it any answer, or redirect the request...

[silverbytes]

Allowed LinksManager since it seems to be the most used and could be legitimate linkchecker. My doubt is about Linkwalker, can't find solid info about it.

[GaryK]

Sticky me if you need the URL to their site. According to their website they seem to check links on sites on behalf of clients. Wish I could tell you more but I don't see it very often in my logs.