homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Googlebot crawling patterns - 0 bytes transfered "Get" requests

 11:36 pm on Aug 9, 2006 (gmt 0)

I launched a new site in late Feb.
It doesn't have a google site map but it is being crawled to its deepest level.

1. Would a site map really make any difference?

2. The number of pages being crawled nightly has been increasing but it really jumped last night to over 1000 but they are all zero bytes transfered get requests. Is Googlebot making its own site map so to speak to come back later for a full crawl?

3. These days I don't have a clue about what to think about Googlebot and its methods. Can anyone enlighten me? Does this sound good or bad?



 1:05 am on Aug 10, 2006 (gmt 0)

Transfer of zero bytes means google is not indexing content. Are you getting any 500 errors in your webstats?


 2:46 am on Aug 10, 2006 (gmt 0)

"Are you getting any 500 errors in your webstats?"

No 500 errors, all pages get a 200 response.

Googlebot is requesting pages in random order, not by going from link to link or page to page, as if they have been there before.


 3:29 am on Aug 10, 2006 (gmt 0)

Transfer of 0 bytes with a 200-OK status means that Google did a Conditional GET, with an If-modified-since HTTP request header, and your server responded with zero bytes because the Last-modified date on the file indicated that Google's cached copy was still current. Therefore, it was unnecessary to transfer the same 'old' content.

The elimination of unnecessary bandwidth and server resource usage is precisely the purpose of the Conditional GET request. You will also see some clients use HEAD requests for the same purpose, although a CGET is more efficient.

In short, don't worry about it. All is well.



 3:47 am on Aug 10, 2006 (gmt 0)

Thanks Jim,

I didn't know what to think. So these are the pages they have been getting already, just checking for changes. Now if I could get them to crawl the rest of the site, as I said they have been going to the deepest level, just not all of it........

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved