Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

GWT - googlebot crawl speed reporting


Oliver Henniges

10:40 am on May 15, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Today for the first time I noticed that google in webmaster central shows detailed information on crawl speed, crawl activity and mass of downloaded data.

Has this been disussed before here?
If not, I have two questions:

1) googlebot alone seems to be responsible for some 5-10% of my overall traffic volume. Is this normal?

2) The average crawl-speed lies around some 1000 milliseconds per page. I think this is quite OK? But there were some peaks with up to 4000 milliseconds. What are the recommended benchmarks concerning eg. adwords-quality-score?


4:00 am on May 16, 2008 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

For question #1, if that's 5% to 10% of total bandwidth, it seems a bit extreme. If it's 5% to 10% of all page requests and many times the file request gets a 304 response with little bandwidth involved, then it sounds OK to me.

For question #2, I'm not much of a server benchmark guy - sorry. Some server benchmarking I've seen indicates that over 1,500 ms generates a lot of user complaint. As a general rule, I've found that Adwords QS imrpoves tremendously as load speed goes up, and I'm beginning to suspect that the same is a factor in organic ranking. And page-load speed involves a lot more than server response time.

Maybe someone else has some idea about server benchmarking?


4:09 am on May 16, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Hi Oliver,

Not sure if crawl speed can be related to load time for AW quality scoring.

Google AdWords says it's slow if it's the regional average plus 3 seconds.




6:39 am on May 16, 2008 (gmt 0)

5+ Year Member

Google in their webmaster guidelines advice:

"Make sure your web server supports the If-Modified-Since HTTP header. This feature allows your web server to tell Google whether your content has changed since we last crawled your site. Supporting this feature saves you bandwidth and overhead."

Maybe its time to implement that.


Oliver Henniges

10:00 am on May 17, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

This 304 thing is new to me and sounds interesting. Thanks.

I assume it has nothing to do with the date-last-modified-entry on harddrive? I use a lot of scripting and even my .html-pages are parsed as php-files. So it wouldn't make sense to deliver a 304 according to datelastmodified if some of the scripts involved have undergone changes meanwhile. And I think I'm not the only one, who does so.

I have a managed server on a well-established larger hosting-company and I assume they have implemented their own cache-system in order to reduce processor capacities, thereby delivering a 304 by default whereever appropriate. I might try to find out about this exactly via mail, but maybe you know, how this is generally handled?

I took a closer look at the figures:

The overall traffic in megabytes amounts to 2-2,5% for the figures given in GWT. As for the number of pageviews googlebot is crawling approximetely one third of my pages every day, which amounts to 5% of all pageviews. I think the difference as compared to traffic volume results from the fact that googlebot does not request images?

The average load-speed lies around 1200 milliseconds and the GWT specifies this as complete pageload-time, not server response-time. I think I can live with that.

I still use a table-based design, but with a php-flush() command after the header-table, which thus loads very quickly. Due to a varying amount of db-requests (up to 40 products per page), some pages (tables!) of course take a bit longer to load but we never had any visitor-complaints concerning load-speed whatsoever.

> I've found that Adwords QS imrpoves tremendously as load speed goes up, and I'm beginning to suspect that the same is a factor in organic ranking.

This is what I also suspect and it would be very interesting to get some details on this.

> regional average plus 3 seconds.

I think I'm far better than that.

I hope the figures I gave might help others to narrow down things a bit. So far two questions remain for me

1) what the hell is "regional average"?
2) Since when does GWT show these graphics? I think the last time I visited that page before was in autumn last year.


Featured Threads

Hot Threads This Week

Hot Threads This Month