Welcome to WebmasterWorld Guest from 188.8.131.52
I block Xenu.
If they don't give you a referer, it seems they are hiding.
You should read "why" on this one - it's on the web site. Give Xenu cautious approval. :)
No, it doesn't follow robots.txt - linkcheckers don't. They are not bots.
One even slaps it with a 403, which strikes me as overkill-- especially in a site that doesn't appear to have a robots.txt at all. (You're supposed to be able to see it on demand, aren't you?)
I do show only the robots.txt permissions for just your user agent to avoid people from attempting to change the user agent to something I allow and test my validation.
How often would one use Xenu to check for broken links in a day? I have the same IP coming in 6 times today, just hitting the home page. This is the second day for this. Do you have to check it that often?
Presumably you've got a fallback for UAs that your server has not previously met-- even if it's a flat
If you're prepared to camouflage your UA to go places you're not supposed to, why bother to ask permission in the first place?