Page Speed measures MUCH more than server response to googlebot's crawl. My sense of it is that the weight placed on page speed is currently rather small, but in the future it might well become much greater.
Crawling itself takes only a very small part of Google's processing overhead. Matt Cutts talked about this recently - I believe it was in one of the Webmaster Help videos, and it rings true to me. It's the storage and indexing that is a big deal, and your page speed has no effect on that major chunk.