Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Mozilla Googlebot and robots.txt

Mozilla Googlebot and robots.txt

         

jrs_66

3:57 pm on Feb 22, 2006 (gmt 0)

10+ Year Member



Consistantly, when Mozilla Googlebot request my robots.txt file, my site log is showing a 301 response code. This has been going on for several weeks. It does not happen with Googlebot 2.1, slurp, or MSNbot. I have no idea why, or how, this could happen.

Anyone seeing this? Any ideas as to how I could stop it?

abates

11:21 pm on Feb 22, 2006 (gmt 0)

10+ Year Member



Are you redirecting example.com to www.example.com or vice versa? It could be requesting example.com/robots.txt and getting a redirect response...

jrs_66

12:30 am on Feb 23, 2006 (gmt 0)

10+ Year Member



I think that's the answer, thanks! Interesting part is that I've yet to see Moz googlebot not do this.