joined:May 24, 2005
One entry in my robots.txt
is shown below, yet today, two Google IP addresses (184.108.40.206, 220.127.116.11) visited [u]only[/u] two files in a directory named /jscript/ .
There was no Google-related U-A in the log entry (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:18.104.22.168) Gecko/20060909 Firefox/22.214.171.124)
, but shouldn't Google be adhering to the below rules regardless of the IP or U-A that they use?
Thanks in advance.