Page is a not externally linkable
mrMister - 6:31 pm on May 4, 2005 (gmt 0)
I agree with you there. But its no different to many other things that some of these "top end" firewalls "protect" you against. They have to justify the extreme cost of these devices somehow!
Blocking any UA (spider or browser) that accesses robots.txt is surely not a solution.