In my surver log, i found googlebot is initially getting 301, when asking for my bots.txt.
I do have in my .htaccess a redirect from non-www to www version to avoid cannonical issue.
I set my preferred domain as www in WMT from last one year and this does not happend before. Log excerpts is ,
66.249.73.xx - "GET /robots.txt HTTP/1.1" 301 329 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.xx - "GET /robots.txt HTTP/1.1" 200 502 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.xx - "GET / HTTP/1.1" 301 319 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.xx - "HEAD / HTTP/1.1" 200 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Any idea, what is happening here? It did not happened before.
As