triggerfinger - 4:59 pm on Sep 17, 2012 (gmt 0)
I'm seeing some seriously strange stuff in our log files.
After receiving warning in GMT about robots.txt inaccessible, we checked the server logs and are seeing the following:
188.8.131.52 www.example.com - [16/Sep/2012:12:21:54 -0400] "GET /exampleproducts/product-2012.htmlrobots.txt HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
Any idea why Google would request incorrect URLs like this? Anyone seeing anything similar?