Forum Moderators: goodroi
User-agent: Xenu's Link Sleuth 1.1c
Disallow: /User-agent: Xenu's
Disallow: /User-agent: Xenu Link Sleuth 1.2a
Disallow: /User-agent: Xenu Link Sleuth 1.2b
Disallow: /User-agent: Xenu Link Sleuth 1.2c
Disallow: /User-agent: Xenu Link Sleuth 1.2d
Disallow: /User-agent: Xenu Link Sleuth 1.2e
Disallow: /User-agent: Xenu Link Sleuth 1.2f
Disallow: /
I have version 1.d and can run it on my site just fine. :(
It's a new web forum and already has 7,000+ links. I really don't need or want Xenu going through my domain.
I spotted Xenu Link Sleuth 1.2e in my logs which is why I looked into Xenu again.
Thing is- when I try to access webmasterworld via Xenu 1.d it doesn't work. I've copied Webmasterworld's robots.txt and given proper credit, so I am not sure what the problem is. Any ideas?
TIA
WB
Configure your web server to block it, based on the agent field. For Apache you use RewriteCond/RewriteRule in .htaccess or httpd.conf. Find details in the related forum, or in server docs.
But, be cautious, because blocking a link-checking tool, is a good way of causing people to remove their "dead links" to your site.
Occasionally I get a 403 error in the report with a site, when I do a manual check I usally find the site is fine, so they probably don't allow that program on their site: wich they have every right to!
Make sure you issue a 403 access denied response and NOT a 404 Not Found to that user agent.
People using the app to check their links will probably interprete a 403 as that you banned that specific user agent and they will check manually. When they see a 404 they will probably not check and remove the link.
I have no links as this is a brand new domain.
I'm on a Windows server so cannot use htaccess
Not a big deal- just irksome.
TIA
WB
I'm on a Windows server so cannot use htaccess
Take a look at ISAPI Rewrite Lite:
isapirewrite.com/
or one of the other IIS rewrite solutions.
I hope to get a dedicated soon.
Regards,
WB
