Welcome to WebmasterWorld Guest from

Forum Moderators: phranque

Message Too Old, No Replies

Best way to prevent bots from following links

anything better than redirect.xyz exluded in robots.txt



12:57 pm on May 20, 2004 (gmt 0)

10+ Year Member

If I'd like to make sure no bot is following links my (maybe a little simple) solution would be:

- use a redirect-file like red.cgi
- put it in some folder like domain.com/admin/red.cgi
- exlude folder /admin/ in robots.txt

Question is if this wouldn't draw any unappreciated attention...

Would you use anything more sophisticated?


5:30 am on May 21, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Do you mean legitimate bots, or spambots? For legitimate bots, the robots.txt should work, or meta robots nofollow.

For spambots, I'd try javascript links. No guarantees, though. Some of them will pull out anything that looks like a url.


6:07 am on May 21, 2004 (gmt 0)

10+ Year Member

>> legitimate bots



Featured Threads

Hot Threads This Week

Hot Threads This Month