themoff - 10:14 pm on Jul 12, 2001 (gmt 0)

I am putting together a robots.txt for my site, and have seen many example ones including UAs such as EmailSyphon. My question is - what's the point? Maybe I'm missing something here, but surely these spiders that are not friendly (either email harvesters, or caching copies of the site etc) can choose to ignore robots.txt? And even if they obey it, surely they could just alter their UA? Is the situation not that it is purely a voluntary standard, and any person writing a spider with the potential to annoy site owners will just build in the capability to either ignore the file, or to mutate UA to an acceptable form?
Cheers, Robin