joined:Apr 8, 2008
I'm seeing some seriously strange stuff in our log files.
After receiving warning in GMT about robots.txt inaccessible, we checked the server logs and are seeing the following:
18.104.22.168 www.example.com - [16/Sep/2012:12:21:54 -0400] "GET /exampleproducts/product-2012.htmlrobots.txt HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
Any idea why Google would request incorrect URLs like this? Anyone seeing anything similar?