Page is a not externally linkable
WarmGlow - 6:27 am on Mar 20, 2004 (gmt 0)
I will use the robots.txt exclusion standard to disallow .js files and keep a close watch on this issue. I intend to wait for a conclusive guilty verdict before I serve 403's to Googlebot/Test.
HTTP_USER_AGENT "Googlebot/Test" requested two objects from one of my domains, robots.txt and a JavaScript file. I do not mind anyone looking at my JavaScript files but I do not want them indexed or cached by a search engine. This does not seem to serve any useful purpose. 64.68.89.156 - - [19/Mar/2004:12:30:07 -0500] "GET /robots.txt HTTP/1.1" 200 1412 "-" "Googlebot/Test" 64.68.89.156 - - [19/Mar/2004:12:30:07 -0500] "GET /01.js HTTP/1.1" 200 152 "-" "Googlebot/Test"