Let me give an idea of why they can use toolbar data to assess site speed ... They Can algorithmically determine if the average load time, template, graphic size, etc. relating to a site is consistent for GoogleBot and relate that to the time a site takes to load relative to the time other sites take to load in the same type niche for visitors, via One Page per site, but they can't do that with Everything relating to a site.
EG A user with the tool bar installed clicks on one page from one site in the results, then clicks on another page from a different site in the results from the same query ... They can make a determination on speed from that.
Because, they can tell:
The template's the same site-wide, the average graphic size is the same site-wide, the server is the same site wide, they sent the visitor to an 'average page' on each site, etc. ... So, if it takes user N452 on connection X 3.2 seconds to download a page from Site A and 9.4 seconds to download a page from Site B, it's Very Safe to assume, overall Site A is roughly 3 times faster than Site B, but that's not a 'conclusion' you can draw for everything 'site wide' from a couple of single page visits by a single user.