| Best way to prevent bots from following links
anything better than redirect.xyz exluded in robots.txt |
Bernie
msg:663984 | 12:57 pm on May 20, 2004 (gmt 0) |
If I'd like to make sure no bot is following links my (maybe a little simple) solution would be: - use a redirect-file like red.cgi
- put it in some folder like domain.com/admin/red.cgi
- exlude folder /admin/ in robots.txt Question is if this wouldn't draw any unappreciated attention... Would you use anything more sophisticated?
|
mcavic
msg:663985 | 5:30 am on May 21, 2004 (gmt 0) |
Do you mean legitimate bots, or spambots? For legitimate bots, the robots.txt should work, or meta robots nofollow. For spambots, I'd try javascript links. No guarantees, though. Some of them will pull out anything that looks like a url.
|
Bernie
msg:663986 | 6:07 am on May 21, 2004 (gmt 0) |
>> legitimate bots yes.
|
|
|
