Page is a not externally linkable
BigDave - 3:28 am on Apr 30, 2003 (gmt 0)
The point I was trying to make is that you should not consider robots.txt exclusions, CSS or JS tricks to keep your site safe.
They don't have to review them manually. The algorithm would be part of the browser. The *only* reason for human involvment is so that it doesn't have to abide by robots.txt, and to possibly check out borderline sites.