Page is a not externally linkable
kryton - 7:35 pm on Mar 10, 2003 (gmt 0)
User-agent: * Will that stop it? It also seems to be going through all my cgi files, something it has not done for a while.
I only link to my adserver via JavaScript, and right now Google is killing my site by making loads of requests to it. I've added the following to my robots.txt
Disallow: /adclick.cgi
Disallow: /adfetch.cgi