Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Google, robots, disallow


cyberdyne - 3:00 pm on May 3, 2011 (gmt 0)


One entry in my robots.txt is shown below, yet today, two Google IP addresses (64.233.172.18, 74.125.75.17) visited [u]only[/u] two files in a directory named /jscript/ .

There was no Google-related U-A in the log entry (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7), but shouldn't Google be adhering to the below rules regardless of the IP or U-A that they use?

User-agent: *
Disallow: /j


Thanks in advance.


Thread source:: http://www.webmasterworld.com/robots_txt/4307351.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com