homepage Welcome to WebmasterWorld Guest from 184.72.82.126
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Inferences from Googlebot Page Download Stats in WMT
jerednel




msg:4331730
 7:18 pm on Jun 27, 2011 (gmt 0)

Let me preface this post with a note from the Webmaster Central article that states that "Googlebot may crawl more than the first 100KB of text." [url]https://www.google.com/support/webmasters/bin/answer.py?answer=158587&hl=en[/url]

I take this to mean that the amount of text crawled by Googlebot varies from site to site based on a number of factors.

I also take this to mean that for sites that may be beyond a 100Kb file size, that some text is not read by Googlebot.

In Webmaster Tools > Diagnostics > Crawl Stats we are given a few graphs, 2 of which are:

    Pages Crawled Per Day

    Kilobytes downloaded per day


Am I correct to assume that: (Kb downloaded per day) / (Pages Crawled Per Day) = Avg Kb downloaded per Page ?

And that all important content and links should be placed before this limit is reached?

Or am I delusional? Thanks!

 

Chrispcritters




msg:4331733
 7:41 pm on Jun 27, 2011 (gmt 0)

I believe that you are taking the note in a FAQ about the "fetch as googlebot" out of context.

jerednel




msg:4331735
 7:49 pm on Jun 27, 2011 (gmt 0)

The note in parenthesis that they have is speaking to Googlebot itself, not the Fetch as Googlebot feature. To say that they "may" fetch more than 100KB is somewhat ambiguous.

tedster




msg:4331750
 8:16 pm on Jun 27, 2011 (gmt 0)

Yes, the amount of crawling certainly varies from site to site, based on a complex calculation.

You will notice that googlebot does not crawl every page with the same frequence, not even on any one website. This means that taking the average, as you are doing, will not give you anything all that meaningful - because the average will be weighted by the specific URLs that were crawled.

And that all important content and links should be placed before this limit is reached?

"Before" doesn't have any context here. Google doesn't start on the home page, then "click" on every link etc. Instead googlebot downloads a page and records the URLs that it finds there in a back end database. All the recorded URLs are then put into a prioritized crawling queue by a complex algorithm. Everytime googlebot goes out crawling, it gets marching orders from that crawling queue.

That's what I mean by "before" not having any context. Just make your pages the minimum-optimum needed for performing their function for visitors.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved