Welcome to WebmasterWorld Guest from 23.22.46.195

Forum Moderators: DixonJones & mademetop

Googlebot downloading pages twice

Googlebot seems to download pages twice

   
12:27 pm on Sep 29, 2009 (gmt 0)

5+ Year Member



I have been recently analyzing various access logs and can see many entries where Googlebot is downloading the same page twice in succession. The following is an example showing two downloads of [at-autos.example.com...]

66.249.67.214 - - [29/Sep/2009:13:47:22 +1000] "GET /honda HTTP/1.1" 200 10502 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.214 - - [29/Sep/2009:13:47:24 +1000] "GET /honda HTTP/1.1" 200 60775 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

My understanding is that the number immediately following the HTTP response code (200 in both lines above) is the number of bytes downloaded. For each pair of downloads, the first number is always (as far as I've checked) smaller than the second.

Does this imply that the first request is being interrupted and the subsequent attempt probably isn't?

Any other ideas what might be going on?

Thanks in anticipation for your help.

[edited by: bill at 9:24 am (utc) on Oct. 2, 2009]
[edit reason] Use example.com [/edit]

8:28 pm on Sep 29, 2009 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Do you have multiple domains pointing to the same website?

I have seen Google crawl each domain I have on one server which were all pointed to the same website.

This is what your log snippet looks like to me.

9:49 am on Oct 1, 2009 (gmt 0)

5+ Year Member



I don't have multiple domains pointing to the same website, but the www version resolves to the same site. What I mean is:

[at-autos.example.com...] and
[at-autos.example.com...]

are served by the same pages, however, I have only added the one without the www prefix to webmaster tools and provided a site map for.

[edited by: Receptional at 8:42 am (utc) on Oct. 2, 2009]
[edit reason] Examplified [/edit]

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month