GaryK - 12:31 am on Nov 4, 2002 (gmt 0) [edited by: GaryK at 12:36 am (utc) on Nov. 4, 2002]
Check the site in my profile. One of the files I offer for download is a regularly updated robots.txt. For robots that don't obey robots.txt you can check the [Website Strippers] section of my browscap.ini file for the user agents I consider a nusiance.
[edited by: GaryK at 12:36 am (utc) on Nov. 4, 2002]