ratboy - 8:59 pm on Apr 9, 2003 (gmt 0)

This is really useful info, I've been wondering how much energy to put out in attempting to block spiders, this has given me enough to make an educated move. Too bad that the way for spider programmers to bypass this htaccess method is so ridiculously easy, but it seems like these techniques will help at least over the short term, thanks especially to toolman and superman, you guys really put out some good stuff, saves us all a lot of work, and many hours of pointless trial and error.