kryton - 2:29 am on Mar 11, 2003 (gmt 0)

projectphp as you can see the googlebot requested robots.txt just before it crawled the cgi page. The only thing I can think of is that there is more than one instance of a googlebot running per IP, and the request made to the cgi page was from a googlebot instance which requested robots.txt before I updated it.

Good idea to change my cgi pages to like you suggested with the <meta name="robots" content="noindex,follow" />