homepage Welcome to WebmasterWorld Guest from 54.197.183.230
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

    
Best way to prevent bots from following links
anything better than redirect.xyz exluded in robots.txt
Bernie

10+ Year Member



 
Msg#: 3356 posted 12:57 pm on May 20, 2004 (gmt 0)

If I'd like to make sure no bot is following links my (maybe a little simple) solution would be:

- use a redirect-file like red.cgi
- put it in some folder like domain.com/admin/red.cgi
- exlude folder /admin/ in robots.txt

Question is if this wouldn't draw any unappreciated attention...

Would you use anything more sophisticated?

 

mcavic

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3356 posted 5:30 am on May 21, 2004 (gmt 0)

Do you mean legitimate bots, or spambots? For legitimate bots, the robots.txt should work, or meta robots nofollow.

For spambots, I'd try javascript links. No guarantees, though. Some of them will pull out anything that looks like a url.

Bernie

10+ Year Member



 
Msg#: 3356 posted 6:07 am on May 21, 2004 (gmt 0)

>> legitimate bots

yes.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved