homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

GWT - googlebot crawl speed reporting
Oliver Henniges

 10:40 am on May 15, 2008 (gmt 0)

Today for the first time I noticed that google in webmaster central shows detailed information on crawl speed, crawl activity and mass of downloaded data.

Has this been disussed before here?
If not, I have two questions:

1) googlebot alone seems to be responsible for some 5-10% of my overall traffic volume. Is this normal?

2) The average crawl-speed lies around some 1000 milliseconds per page. I think this is quite OK? But there were some peaks with up to 4000 milliseconds. What are the recommended benchmarks concerning eg. adwords-quality-score?



 4:00 am on May 16, 2008 (gmt 0)

For question #1, if that's 5% to 10% of total bandwidth, it seems a bit extreme. If it's 5% to 10% of all page requests and many times the file request gets a 304 response with little bandwidth involved, then it sounds OK to me.

For question #2, I'm not much of a server benchmark guy - sorry. Some server benchmarking I've seen indicates that over 1,500 ms generates a lot of user complaint. As a general rule, I've found that Adwords QS imrpoves tremendously as load speed goes up, and I'm beginning to suspect that the same is a factor in organic ranking. And page-load speed involves a lot more than server response time.

Maybe someone else has some idea about server benchmarking?


 4:09 am on May 16, 2008 (gmt 0)

Hi Oliver,

Not sure if crawl speed can be related to load time for AW quality scoring.

Google AdWords says it's slow if it's the regional average plus 3 seconds.




 6:39 am on May 16, 2008 (gmt 0)

Google in their webmaster guidelines advice:

"Make sure your web server supports the If-Modified-Since HTTP header. This feature allows your web server to tell Google whether your content has changed since we last crawled your site. Supporting this feature saves you bandwidth and overhead."

Maybe its time to implement that.


Oliver Henniges

 10:00 am on May 17, 2008 (gmt 0)

This 304 thing is new to me and sounds interesting. Thanks.

I assume it has nothing to do with the date-last-modified-entry on harddrive? I use a lot of scripting and even my .html-pages are parsed as php-files. So it wouldn't make sense to deliver a 304 according to datelastmodified if some of the scripts involved have undergone changes meanwhile. And I think I'm not the only one, who does so.

I have a managed server on a well-established larger hosting-company and I assume they have implemented their own cache-system in order to reduce processor capacities, thereby delivering a 304 by default whereever appropriate. I might try to find out about this exactly via mail, but maybe you know, how this is generally handled?

I took a closer look at the figures:

The overall traffic in megabytes amounts to 2-2,5% for the figures given in GWT. As for the number of pageviews googlebot is crawling approximetely one third of my pages every day, which amounts to 5% of all pageviews. I think the difference as compared to traffic volume results from the fact that googlebot does not request images?

The average load-speed lies around 1200 milliseconds and the GWT specifies this as complete pageload-time, not server response-time. I think I can live with that.

I still use a table-based design, but with a php-flush() command after the header-table, which thus loads very quickly. Due to a varying amount of db-requests (up to 40 products per page), some pages (tables!) of course take a bit longer to load but we never had any visitor-complaints concerning load-speed whatsoever.

> I've found that Adwords QS imrpoves tremendously as load speed goes up, and I'm beginning to suspect that the same is a factor in organic ranking.

This is what I also suspect and it would be very interesting to get some details on this.

> regional average plus 3 seconds.

I think I'm far better than that.

I hope the figures I gave might help others to narrow down things a bit. So far two questions remain for me

1) what the hell is "regional average"?
2) Since when does GWT show these graphics? I think the last time I visited that page before was in autumn last year.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved