WMT user real uses with the Google Toolbar or Chrome browser installed. Their load time will include DNS lookup time and network latency that you may not see measuring your site yourself locally. Try some other browser testing sites and see if you site is slow to load from a variety of cities around the world.
WMT also may not be measuring for many users. It will say either "These estimates are of high accuracy (more than 1000 data points)." or "These estimates are of low accuracy (fewer than 100 data points).". If it is the latter, then you may not need to pay much attention to it, especially if the graph varies widely. It may just have happened to hit a few users from North Korea browsing on 2400 baud modems.