Page is a not externally linkable
cyberdyne - 3:00 pm on May 3, 2011 (gmt 0)
One entry in my robots.txt is shown below, yet today, two Google IP addresses (64.233.172.18, 74.125.75.17) visited [u]only[/u] two files in a directory named /jscript/ .
There was no Google-related U-A in the log entry (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7), but shouldn't Google be adhering to the below rules regardless of the IP or U-A that they use?
User-agent: *
Disallow: /j
Thanks in advance.