Welcome to WebmasterWorld Guest from 54.160.221.82

Forum Moderators: phranque

Message Too Old, No Replies

Best way to prevent bots from following links

anything better than redirect.xyz exluded in robots.txt

     
12:57 pm on May 20, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:June 1, 2002
posts:382
votes: 0


If I'd like to make sure no bot is following links my (maybe a little simple) solution would be:

- use a redirect-file like red.cgi
- put it in some folder like domain.com/admin/red.cgi
- exlude folder /admin/ in robots.txt

Question is if this wouldn't draw any unappreciated attention...

Would you use anything more sophisticated?

5:30 am on May 21, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 31, 2003
posts:1316
votes: 0


Do you mean legitimate bots, or spambots? For legitimate bots, the robots.txt should work, or meta robots nofollow.

For spambots, I'd try javascript links. No guarantees, though. Some of them will pull out anything that looks like a url.

6:07 am on May 21, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:June 1, 2002
posts:382
votes: 0


>> legitimate bots

yes.