Welcome to WebmasterWorld Guest from 54.196.214.35

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Googlebot downloading pages twice

Googlebot seems to download pages twice

     
12:27 pm on Sep 29, 2009 (gmt 0)

New User

5+ Year Member

joined:June 11, 2008
posts:15
votes: 0


I have been recently analyzing various access logs and can see many entries where Googlebot is downloading the same page twice in succession. The following is an example showing two downloads of [at-autos.example.com...]

66.249.67.214 - - [29/Sep/2009:13:47:22 +1000] "GET /honda HTTP/1.1" 200 10502 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.214 - - [29/Sep/2009:13:47:24 +1000] "GET /honda HTTP/1.1" 200 60775 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

My understanding is that the number immediately following the HTTP response code (200 in both lines above) is the number of bytes downloaded. For each pair of downloads, the first number is always (as far as I've checked) smaller than the second.

Does this imply that the first request is being interrupted and the subsequent attempt probably isn't?

Any other ideas what might be going on?

Thanks in anticipation for your help.

[edited by: bill at 9:24 am (utc) on Oct. 2, 2009]
[edit reason] Use example.com [/edit]

8:28 pm on Sept 29, 2009 (gmt 0)

Administrator

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month

joined:Jan 14, 2004
posts:852
votes: 0


Do you have multiple domains pointing to the same website?

I have seen Google crawl each domain I have on one server which were all pointed to the same website.

This is what your log snippet looks like to me.

9:49 am on Oct 1, 2009 (gmt 0)

New User

5+ Year Member

joined:June 11, 2008
posts:15
votes: 0


I don't have multiple domains pointing to the same website, but the www version resolves to the same site. What I mean is:

[at-autos.example.com...] and
[at-autos.example.com...]

are served by the same pages, however, I have only added the one without the www prefix to webmaster tools and provided a site map for.

[edited by: Receptional at 8:42 am (utc) on Oct. 2, 2009]
[edit reason] Examplified [/edit]