homepage Welcome to WebmasterWorld Guest from 54.146.175.204
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Googlebot downloading pages twice
Googlebot seems to download pages twice
dugald

5+ Year Member



 
Msg#: 3997632 posted 12:27 pm on Sep 29, 2009 (gmt 0)

I have been recently analyzing various access logs and can see many entries where Googlebot is downloading the same page twice in succession. The following is an example showing two downloads of [at-autos.example.com...]

66.249.67.214 - - [29/Sep/2009:13:47:22 +1000] "GET /honda HTTP/1.1" 200 10502 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.214 - - [29/Sep/2009:13:47:24 +1000] "GET /honda HTTP/1.1" 200 60775 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

My understanding is that the number immediately following the HTTP response code (200 in both lines above) is the number of bytes downloaded. For each pair of downloads, the first number is always (as far as I've checked) smaller than the second.

Does this imply that the first request is being interrupted and the subsequent attempt probably isn't?

Any other ideas what might be going on?

Thanks in anticipation for your help.

[edited by: bill at 9:24 am (utc) on Oct. 2, 2009]
[edit reason] Use example.com [/edit]

 

Ocean10000

WebmasterWorld Administrator 10+ Year Member



 
Msg#: 3997632 posted 8:28 pm on Sep 29, 2009 (gmt 0)

Do you have multiple domains pointing to the same website?

I have seen Google crawl each domain I have on one server which were all pointed to the same website.

This is what your log snippet looks like to me.

dugald

5+ Year Member



 
Msg#: 3997632 posted 9:49 am on Oct 1, 2009 (gmt 0)

I don't have multiple domains pointing to the same website, but the www version resolves to the same site. What I mean is:

[at-autos.example.com...] and
[at-autos.example.com...]

are served by the same pages, however, I have only added the one without the www prefix to webmaster tools and provided a site map for.

[edited by: Receptional at 8:42 am (utc) on Oct. 2, 2009]
[edit reason] Examplified [/edit]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved