homepage Welcome to WebmasterWorld Guest from 54.242.126.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Can't seem to block Xenu
WebBender

10+ Year Member



 
Msg#: 318 posted 8:19 am on Mar 10, 2004 (gmt 0)

Have the following in robots.txt in root folder:

User-agent: Xenu's Link Sleuth 1.1c
Disallow: /

User-agent: Xenu's
Disallow: /

User-agent: Xenu Link Sleuth 1.2a
Disallow: /

User-agent: Xenu Link Sleuth 1.2b
Disallow: /

User-agent: Xenu Link Sleuth 1.2c
Disallow: /

User-agent: Xenu Link Sleuth 1.2d
Disallow: /

User-agent: Xenu Link Sleuth 1.2e
Disallow: /

User-agent: Xenu Link Sleuth 1.2f
Disallow: /

I have version 1.d and can run it on my site just fine. :(

It's a new web forum and already has 7,000+ links. I really don't need or want Xenu going through my domain.

I spotted Xenu Link Sleuth 1.2e in my logs which is why I looked into Xenu again.

Thing is- when I try to access webmasterworld via Xenu 1.d it doesn't work. I've copied Webmasterworld's robots.txt and given proper credit, so I am not sure what the problem is. Any ideas?

TIA

WB

 

Robert Thivierge

10+ Year Member



 
Msg#: 318 posted 10:18 am on Mar 10, 2004 (gmt 0)

Xenu never reads robots.txt, so don't bother guessing a User-agent (it even ignores *).

Configure your web server to block it, based on the agent field. For Apache you use RewriteCond/RewriteRule in .htaccess or httpd.conf. Find details in the related forum, or in server docs.

But, be cautious, because blocking a link-checking tool, is a good way of causing people to remove their "dead links" to your site.

DoppyNL

10+ Year Member



 
Msg#: 318 posted 10:31 am on Mar 10, 2004 (gmt 0)

I use the app to check both my internal links and external links.
External site's will only get 1 request from the app from me.

Occasionally I get a 403 error in the report with a site, when I do a manual check I usally find the site is fine, so they probably don't allow that program on their site: wich they have every right to!

Make sure you issue a 403 access denied response and NOT a 404 Not Found to that user agent.
People using the app to check their links will probably interprete a 403 as that you banned that specific user agent and they will check manually. When they see a 404 they will probably not check and remove the link.

WebBender

10+ Year Member



 
Msg#: 318 posted 9:36 pm on Mar 10, 2004 (gmt 0)

Hey guys,

I have no links as this is a brand new domain.

I'm on a Windows server so cannot use htaccess

Not a big deal- just irksome.

TIA

WB

bcolflesh

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 318 posted 9:59 pm on Mar 10, 2004 (gmt 0)

I'm on a Windows server so cannot use htaccess

Take a look at ISAPI Rewrite Lite:

isapirewrite.com/

or one of the other IIS rewrite solutions.

WebBender

10+ Year Member



 
Msg#: 318 posted 12:55 am on Mar 12, 2004 (gmt 0)

Thanks- I'm aware of it, but am on a Virtual Server.

I hope to get a dedicated soon.

Regards,

WB

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved