awoyo - 10:29 pm on Jul 12, 2001 (gmt 0) Happy hunting! Jim
It's my experience that robots.txt is only effective for spiders who voluntarily follow the exclusion protocol, therefore, you are correct. When it comes to rogue spiders, robots.txt is of no particular use. However, when it's used as part of the whole; i.e. robots.txt, .htaccess use of mod_access or mod_rewrite, and finally some scripting measures, it becomes, imho, worthy, of being included into the "system".