homepage Welcome to WebmasterWorld Guest from 23.23.8.131
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
400 Xenu Link Sleuth hits this month
how do I block them.....
Daily Sparring

10+ Year Member



 
Msg#: 2604 posted 2:00 am on Aug 23, 2004 (gmt 0)

What is the proper robots text? Should I use an .htaccess. I will start with robots .txt disallow but I am not sure what to call it.

 

martinibuster

WebmasterWorld Administrator martinibuster us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 2604 posted 2:33 am on Aug 23, 2004 (gmt 0)

Try here [robotstxt.org]. All you ever wanted to know about robots.txt in easy to understand chunks.

DanA

10+ Year Member



 
Msg#: 2604 posted 9:00 am on Aug 23, 2004 (gmt 0)

Xenu, a (dead) link checher will not ask for robots.txt
you may find a list of resources by looking for "web spider traps" or "How to Defeat Bad Web Robots" with google

TheDoctor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2604 posted 9:29 am on Aug 23, 2004 (gmt 0)

have you checkede the referrer? It sounds like someone with a link to your site has been frantically trying to clean up their site.

If it's all from a single referrer, an email asking for a bit of moderation (and deskchecking) might do the trick.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved