Welcome to WebmasterWorld Guest from 54.196.243.192

Message Too Old, No Replies

Googlebot crawling patterns - 0 bytes transfered "Get" requests

     
11:36 pm on Aug 9, 2006 (gmt 0)

New User

5+ Year Member

joined:Mar 4, 2006
posts:22
votes: 0


I launched a new site in late Feb.
It doesn't have a google site map but it is being crawled to its deepest level.

1. Would a site map really make any difference?

2. The number of pages being crawled nightly has been increasing but it really jumped last night to over 1000 but they are all zero bytes transfered get requests. Is Googlebot making its own site map so to speak to come back later for a full crawl?

3. These days I don't have a clue about what to think about Googlebot and its methods. Can anyone enlighten me? Does this sound good or bad?

1:05 am on Aug 10, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 5, 2006
posts:2094
votes: 2


Transfer of zero bytes means google is not indexing content. Are you getting any 500 errors in your webstats?
2:46 am on Aug 10, 2006 (gmt 0)

New User

5+ Year Member

joined:Mar 4, 2006
posts:22
votes: 0


"Are you getting any 500 errors in your webstats?"

No 500 errors, all pages get a 200 response.

Googlebot is requesting pages in random order, not by going from link to link or page to page, as if they have been there before.

3:29 am on Aug 10, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Transfer of 0 bytes with a 200-OK status means that Google did a Conditional GET, with an If-modified-since HTTP request header, and your server responded with zero bytes because the Last-modified date on the file indicated that Google's cached copy was still current. Therefore, it was unnecessary to transfer the same 'old' content.

The elimination of unnecessary bandwidth and server resource usage is precisely the purpose of the Conditional GET request. You will also see some clients use HEAD requests for the same purpose, although a CGET is more efficient.

In short, don't worry about it. All is well.

Jim

3:47 am on Aug 10, 2006 (gmt 0)

New User

5+ Year Member

joined:Mar 4, 2006
posts:22
votes: 0


Thanks Jim,

I didn't know what to think. So these are the pages they have been getting already, just checking for changes. Now if I could get them to crawl the rest of the site, as I said they have been going to the deepest level, just not all of it........