The data is coming from the Google toolbar, and thus you cannot prevent this being recorded short of blocking google tooolbar users.
Personally, I think the GWT speed reports are riddled with errors and oversights. I have one site, for instance, whereby users submit a large volume of data to be processed. This process takes some time, meaning that Google lists speed averages as astronomically high.
The are various other bugs and similar that render many of the reports I see unusable as a measure of speed, let alone a quality measure to be used within a search algorithm.