What is the proper robots text? Should I use an .htaccess. I will start with robots .txt disallow but I am not sure what to call it.
martinibuster
2:33 am on Aug 23, 2004 (gmt 0)
Try here [robotstxt.org]. All you ever wanted to know about robots.txt in easy to understand chunks.
DanA
9:00 am on Aug 23, 2004 (gmt 0)
Xenu, a (dead) link checher will not ask for robots.txt you may find a list of resources by looking for "web spider traps" or "How to Defeat Bad Web Robots" with google
TheDoctor
9:29 am on Aug 23, 2004 (gmt 0)
have you checkede the referrer? It sounds like someone with a link to your site has been frantically trying to clean up their site.
If it's all from a single referrer, an email asking for a bit of moderation (and deskchecking) might do the trick.
