chiyo - 6:14 pm on Jul 6, 2003 (gmt 0)

I'm pretty sure the robots.txt is to enable web site owners to advise which folders not to index, not the folders not to be spidered. So you can respect robots.txt even if you do spider such folders, as long as you dont index them.

if you are worried about bandwidth, there are better ways to solve that problem, but i wouldnt classify scotter as disrespectful bot just because it had a peek!