Page is a not externally linkable
Craven_de_Kere - 4:51 pm on May 5, 2005 (gmt 0)
"> it does not honor robots.txt It is a proxy on behalf of a human, it isn't a bot." It's more than a proxy. It is sending automated requests. Those automated requests do qualify as being a bot. Just as does software for offline viewing of a website. A single human visitor may be the one using it, but automated requests means it *is* a bot.
Brett Wrote: