Page is a not externally linkable
bears5122 - 8:40 pm on Feb 20, 2006 (gmt 0)
The problem with bad bots is only going to get worse with time.
It'll be interesting to see how this is handled over the years as I too believe that robots.txt is outdated. You wonder if we'll hit a day when bot owners must lobby web hosts to allow access to their server. Kind of like an invitation only policy.