I have a website which was hosted in Germany till last month. Then this month I moved it to a US based server. The new host is one of the best and top ranking dedicated hosting provider. Ever since I have moved the site, I have heard people saying site loads slow now. Although I personally haven't seen it because when I open it myself or use some 3rd party website speed checker then it shows excellent speed. But the thing is when I go to Google Webmaster and there is a something called "site performance" under "Labs" then it shows that my site's loading time is increasing. And the home page is taking on average 10 seconds to open completely. But of course when I open website myself or if I ask my friend, we don't notice such delay. In fact this is totally absurd that Google is saying 10 seconds for home page! Upon reading further about site performance I came to know that this is the data that Google collects from Google Toolbar that users have installed. So that means its results could be skewed and totally unreliable!? But two people I talked to said that its not just toolbar but Google also collects this data by other means. Although Google help documents only mention about toolbar. So I want to know what is true? And when you see crawl stats then Google says that it took an average 0.6 seconds to crawl one page. So crawl stats is showing something else while site performance is showing totally different.
Secondly what tools and techniques can I use to really see what the problem is? Shall I look into webalizer or raw access logs? I have also looked into MySQL logs and found some queries which are slowing site performance but not on home page. And that too has only happened twice for a brief period of 1 or 2 minutes when site was not loading at all due to MySQL server being busy. What data shall I look for in logs and stats? If there really is some problem then I want to go to bottom of it.
I have checked YSlow and Google Page Speed (in Firefox) and they do say that there are some problems related to gzipping css and js and some other but I don't think they will affect that much although I will definitely fix them in coming days.
No idea about waterfall graphs will check it out.
That's the main problem because these complains are also coming from business and SEO department and they are based on just Google's site performance tool because I haven't seen any concrete complain as to what exactly is wrong. It's the usual talk that "oh the site is slow...many users have said" but when I ask specifics then no answer.
I am using Apache on CentOS with 64 bit Intel Xeon 8 CPUs.
Yes browser caching is also something which I need to look into but no idea about server caching.