Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

Googlebot crawling patterns - 0 bytes transfered "Get" requests



11:36 pm on Aug 9, 2006 (gmt 0)

5+ Year Member

I launched a new site in late Feb.
It doesn't have a google site map but it is being crawled to its deepest level.

1. Would a site map really make any difference?

2. The number of pages being crawled nightly has been increasing but it really jumped last night to over 1000 but they are all zero bytes transfered get requests. Is Googlebot making its own site map so to speak to come back later for a full crawl?

3. These days I don't have a clue about what to think about Googlebot and its methods. Can anyone enlighten me? Does this sound good or bad?


1:05 am on Aug 10, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member

Transfer of zero bytes means google is not indexing content. Are you getting any 500 errors in your webstats?


2:46 am on Aug 10, 2006 (gmt 0)

5+ Year Member

"Are you getting any 500 errors in your webstats?"

No 500 errors, all pages get a 200 response.

Googlebot is requesting pages in random order, not by going from link to link or page to page, as if they have been there before.


3:29 am on Aug 10, 2006 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Transfer of 0 bytes with a 200-OK status means that Google did a Conditional GET, with an If-modified-since HTTP request header, and your server responded with zero bytes because the Last-modified date on the file indicated that Google's cached copy was still current. Therefore, it was unnecessary to transfer the same 'old' content.

The elimination of unnecessary bandwidth and server resource usage is precisely the purpose of the Conditional GET request. You will also see some clients use HEAD requests for the same purpose, although a CGET is more efficient.

In short, don't worry about it. All is well.



3:47 am on Aug 10, 2006 (gmt 0)

5+ Year Member

Thanks Jim,

I didn't know what to think. So these are the pages they have been getting already, just checking for changes. Now if I could get them to crawl the rest of the site, as I said they have been going to the deepest level, just not all of it........


Featured Threads

Hot Threads This Week

Hot Threads This Month