Welcome to WebmasterWorld Guest from

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

400 Xenu Link Sleuth hits this month

how do I block them.....

2:00 am on Aug 23, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 23, 2003
votes: 0

What is the proper robots text? Should I use an .htaccess. I will start with robots .txt disallow but I am not sure what to call it.
2:33 am on Aug 23, 2004 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
votes: 380

Try here [robotstxt.org]. All you ever wanted to know about robots.txt in easy to understand chunks.
9:00 am on Aug 23, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 29, 2003
votes: 0

Xenu, a (dead) link checher will not ask for robots.txt
you may find a list of resources by looking for "web spider traps" or "How to Defeat Bad Web Robots" with google
9:29 am on Aug 23, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 6, 2003
votes: 0

have you checkede the referrer? It sounds like someone with a link to your site has been frantically trying to clean up their site.

If it's all from a single referrer, an email asking for a bit of moderation (and deskchecking) might do the trick.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members