Robert Thivierge
msg:1525439 | 10:18 am on Mar 10, 2004 (gmt 0) |
Xenu never reads robots.txt, so don't bother guessing a User-agent (it even ignores *). Configure your web server to block it, based on the agent field. For Apache you use RewriteCond/RewriteRule in .htaccess or httpd.conf. Find details in the related forum, or in server docs. But, be cautious, because blocking a link-checking tool, is a good way of causing people to remove their "dead links" to your site.
|
DoppyNL
msg:1525440 | 10:31 am on Mar 10, 2004 (gmt 0) |
I use the app to check both my internal links and external links.
External site's will only get 1 request from the app from me. Occasionally I get a 403 error in the report with a site, when I do a manual check I usally find the site is fine, so they probably don't allow that program on their site: wich they have every right to! Make sure you issue a 403 access denied response and NOT a 404 Not Found to that user agent.
People using the app to check their links will probably interprete a 403 as that you banned that specific user agent and they will check manually. When they see a 404 they will probably not check and remove the link.
|
WebBender
msg:1525441 | 9:36 pm on Mar 10, 2004 (gmt 0) |
Hey guys, I have no links as this is a brand new domain. I'm on a Windows server so cannot use htaccess Not a big deal- just irksome. TIA WB
|
bcolflesh
msg:1525442 | 9:59 pm on Mar 10, 2004 (gmt 0) |
| I'm on a Windows server so cannot use htaccess |
|
Take a look at ISAPI Rewrite Lite: isapirewrite.com/ or one of the other IIS rewrite solutions.
|
WebBender
msg:1525443 | 12:55 am on Mar 12, 2004 (gmt 0) |
Thanks- I'm aware of it, but am on a Virtual Server. I hope to get a dedicated soon. Regards, WB
|
|
