Anyone know how to ban Xenu Link Sleuth?

It makes me feel like there's a peep hole into my web site.

[martinibuster]

Gives me an icky feeling to know that someone's undressing my web site in this manner. Any way to ban them?

I've searched webmasterworld but didn't find a reference to how to lock out them out. Anyone know?

[pageoneresults]

This from Brett's robots.txt file...

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: Xenu's
Disallow: /

[Woz]

martinibuster,

Xenu is a Link Checker that checks both internal and external links for 404's. If someone is just checking one link or so then that is not a bad thing and I wouldn't worry about it. They are probably just making sure their link to your page is still OK. Also, in that case I believe it checks the page linked to without first consulting robots.txt but I may be wrong.

However, the savvy user can also use it to harvest links from your site. Robots.txt would help in this case but you also might want to look into a banning script/funcition, depending on what sort of server you are using, if this gets out of hand.

Onya
Woz

[Axacta]

Interesting that you bring this up martinibuster, because just yesterday somebody accessed my site through my WW profile, and then came back later and spidered my whole site with Xenu Link Sleuth. I agree about the icky feeling. I almost feel a sense of betrayal knowing it came through WW. (Of course I don't blame WW.)

[martinibuster]

Thanks!

[chiyo]

We use Xenu Link Sleuth to make sure all our links are still working. Information you put on the Web is public - you have to live with that and I dont see robots doing anything wrong. If the bandwidth is a problem or you feel your content is being indexed for reasons you dont feel are useful to you, you have the option to block any or all spiders through robots.txt or other means.

In our case if Xenu tells us that a URL is no longer available we just delete the link as we dont have time to check it any further. So by blocking this spider, you may lose some existing incoming links if other people are like me.

[martinibuster]

That's an interesting thought, Chiyo.

[victor]

Link Sleuth also builds a site map -- a hierarchical structure from the home page.

I know a couple of visually disabled users who find the Xenu site map much easier to follow as a way into a site than the sometimes over designed graphic flashiness of the site' home page itself.

It would be a shame if they lost that ability.

[richlowe]

I wouldn't touch Xenu Link Sleuth with a ten foot pole, cannot recommend it to anyone and have banned it from my site. Junk software.

Richard Lowe

[highman]

>I wouldn't touch Xenu Link Sleuth with a ten foot pole, cannot recommend it to anyone and have banned it from my site. Junk software.

Why? its a great little tool, why do you slam it so?

[chiyo]

What are your reasons Richard? We have found it a great timesaver and as accurate as commercial offerings for many years. We recommend it highly so are interested in why we shouldn't.

[victor]

Richard, are there currently available link checkers you would recommend? (I checked your site, but several you list are no longer available). Thanks.

[rogerd]

Rich, I'm also curious why you don't like Xenu. I've found it to be a convenient site mapping tool and handy broken link checker. Whenever someone asks me to work on a site, the first thing I do is run Xenu to get a handle on what's out there and how degraded their linkage is. Xenu's site maps leave a bit to be desired, but you can't beat the price.

[Key_Master]

I ban it simply because it is a great tool. Too good! ;)