joined:May 24, 2005
One entry in my robots.txt
is shown below, yet today, two Google IP addresses (22.214.171.124, 126.96.36.199) visited [u]only[/u] two files in a directory named /jscript/ .
There was no Google-related U-A in the log entry (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:188.8.131.52) Gecko/20060909 Firefox/184.108.40.206)
, but shouldn't Google be adhering to the below rules regardless of the IP or U-A that they use?
Thanks in advance.