One entry in my robots.txt
is shown below, yet today, two Google IP addresses (188.8.131.52, 184.108.40.206) visited [u]only[/u] two files in a directory named /jscript/ .
There was no Google-related U-A in the log entry (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:220.127.116.11) Gecko/20060909 Firefox/18.104.22.168)
, but shouldn't Google be adhering to the below rules regardless of the IP or U-A that they use?
Thanks in advance.