homepage Welcome to WebmasterWorld Guest from 54.196.168.78
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Googlebot downloading pages twice
Googlebot seems to download pages twice
dugald




msg:3997634
 12:27 pm on Sep 29, 2009 (gmt 0)

I have been recently analyzing various access logs and can see many entries where Googlebot is downloading the same page twice in succession. The following is an example showing two downloads of [at-autos.example.com...]

66.249.67.214 - - [29/Sep/2009:13:47:22 +1000] "GET /honda HTTP/1.1" 200 10502 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.214 - - [29/Sep/2009:13:47:24 +1000] "GET /honda HTTP/1.1" 200 60775 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

My understanding is that the number immediately following the HTTP response code (200 in both lines above) is the number of bytes downloaded. For each pair of downloads, the first number is always (as far as I've checked) smaller than the second.

Does this imply that the first request is being interrupted and the subsequent attempt probably isn't?

Any other ideas what might be going on?

Thanks in anticipation for your help.

[edited by: bill at 9:24 am (utc) on Oct. 2, 2009]
[edit reason] Use example.com [/edit]

 

Ocean10000




msg:3997992
 8:28 pm on Sep 29, 2009 (gmt 0)

Do you have multiple domains pointing to the same website?

I have seen Google crawl each domain I have on one server which were all pointed to the same website.

This is what your log snippet looks like to me.

dugald




msg:3999148
 9:49 am on Oct 1, 2009 (gmt 0)

I don't have multiple domains pointing to the same website, but the www version resolves to the same site. What I mean is:

[at-autos.example.com...] and
[at-autos.example.com...]

are served by the same pages, however, I have only added the one without the www prefix to webmaster tools and provided a site map for.

[edited by: Receptional at 8:42 am (utc) on Oct. 2, 2009]
[edit reason] Examplified [/edit]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved