joined:May 24, 2005
One entry in my robots.txt
is shown below, yet today, two Google IP addresses (18.104.22.168, 22.214.171.124) visited [u]only[/u] two files in a directory named /jscript/ .
There was no Google-related U-A in the log entry (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:126.96.36.199) Gecko/20060909 Firefox/188.8.131.52)
, but shouldn't Google be adhering to the below rules regardless of the IP or U-A that they use?
Thanks in advance.