Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Googlebot getting 'canonical' redirects for robots.txt

         

marodhum

2:38 pm on Feb 20, 2008 (gmt 0)

10+ Year Member



In my surver log, i found googlebot is initially getting 301, when asking for my bots.txt.
I do have in my .htaccess a redirect from non-www to www version to avoid cannonical issue.
I set my preferred domain as www in WMT from last one year and this does not happend before. Log excerpts is ,
66.249.73.xx - "GET /robots.txt HTTP/1.1" 301 329 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.xx - "GET /robots.txt HTTP/1.1" 200 502 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.xx - "GET / HTTP/1.1" 301 319 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.xx - "HEAD / HTTP/1.1" 200 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Any idea, what is happening here? It did not happened before.
As

tedster

6:10 pm on Feb 20, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It sounds like your htaccess is doing its job, and for whatever reason googlebot has a no-www version of the urlin its crawl queue.

In your Webmaster Tools, is there any negative report under Tools > Analyze Robots.txt?

marodhum

8:04 pm on Feb 20, 2008 (gmt 0)

10+ Year Member



Nope, nothing there. In fact the status showing 200(success).
I forgot to tell, this is happening only in case of robots.txt and index file. For all other pages, googlebot is asking the non www version.
66.249.73.xx - "GET /somepage.html HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

and last but not the least, Teds.. 3 cheers for you.
i read the post [seomoz.org] here.

p.s- Is it possible? that someone link me with the non www version and Gbot while crawling from there asking for the same?

As

tedster

8:09 pm on Feb 20, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, that's very possible.

MadeWillis

8:17 pm on Feb 20, 2008 (gmt 0)

10+ Year Member



Can't you tell Google to read both version as the same in Webmaster Tools?

marodhum

2:35 am on Feb 21, 2008 (gmt 0)

10+ Year Member



MadeWillis, it is not possible in WMT. Instead there is a Set Preferred Domain feature available, through which you can tell Google to associate a particular version, either www or non-www with your site.

As