Site Speed Issues

Poor site speed for one client, not others.

[Ozzy]

Hoping for a pointer here. We provide eCommerce software for a specific industry sector and for that we host our clients sites.
We use Google Cloud for this and the performance seems perfectly fine for all our clients except for one, and we're out of ideas why.
We are using a VM and our platform is PHP with Cloud SQL behind it. All our client sites are on the same VM, using the same database, running the same code.

Some examples of some client websites running perfectly fine are;
[snip]
These all load the home page (and other pages) in less than 3 seconds, and we've tested using pingdom [tools.pingdom.com]

However we have another client website which is taking 15-20 seconds to load the home page, and we cannot see a logical reason for this. The client is claiming it is our framework but with all our other client websites working perfectly fine this cannot be the case.
Running out of ideas so hoping someone out there is able to give some pointers why;
[snip]

Here's hoping :)
Thanks, Richard

[edited by: phranque at 6:35 am (utc) on Apr 9, 2015]

[phranque]

none of the network timeline/waterfall charts the various browsers' web developer tools provide tell you anything useful?

perhaps it's a caching-related headers issue on the one server - have you tried any of the page speed tools?

any significant differences in sizes of page resources?

[Ozzy]

The waterfall on the site speed tests show a longer "wait" for the assets from the problem website than the other fine websites. This does imply our code but the confusing issue is that it is the same code for all the sites, and I don't mean same type of code I mean it quite literally is the exact same physical files.
The asset sites are fine, a few K here and there. The directory paths are fine, and loading the assets on their own independently (not as part of the whole page) loads them in a microsecond. The assets only load slowly when being loaded as part of the entire page, they are fine when loaded individually.

[phranque]

Have you compared the configuration settings of the httpd.conf files?

[5kKate]

Perhaps your client is located in a different geography or has a slower network path? If this is a possibility you can see load time by geography in google analytics. This might be an issue if you have lots of assets that are loading synchronously. You can try bundling them together to reduce requests, and async to parallelize them. If your client's connection is slow you can try reducing the size of the assets using minify and compression.

[Ozzy]

Thanks for the replies so far guys.
All our client sites (about 50 of them) run from one VirtualHost container in the Apache conf file. The VirtualHost is a *:80 entry with no ServerName, it's the default site for the server. Our PHP code then looks at the requested domain to work out what site to display to the browser.
Before anyone says anything, yes I know serving assets through PHP is not the most efficient but this method allows us to meet the business needs. Out of all 50 odd sites running on this platform through the same code all assets for all other sites are being delivered in circa 150ms to the browser except this one website where each asset takes about 1-2 seconds each asset.
This has been tested on a number of website speed test sites, such as pingdom mentioned earlier. These asset speeds are taken from the waterfall of the speed test.

Now to add some more confusion to the mix. If you test any of the assets on their own individually through the same speed test site the asset loads about 100ms Then look at that same image being loaded as part of the full page load and it takes 1-2 seconds. This is only on that one client site, all our other client sites on the same server and code behave consistently and quick.

We've ruled out location by using different speed test websites and selected tests from different locations too. It's a consistent problem.

This has only happened since we moved from our own servers we owned outright but it was time to decommission, and moved our framework to Google Cloud. We replaced our two dedicated MySQL servers (master and slave) with Google Cloud SQL, and we replaced our web server with a Google VM Instance. We have configured Cloud SQL to the highest tier they offer, and the VM is close to the highest tier (CPU load averaging at about 3% to give you an idea). All of our clients have been pleased to see a significant performance improvement (our old servers were about 8 years old and tired), but this one. We pay for the Google god support so have asked them to check server and infrastructure setup, they couldn't see any issues.

Nothing is making any logical sense :(

Due to the number of complaints from this client and the need to pay compensation (SLA) we've created a separate VirtualHost in Apache for this one client with a directory alias to bypass our code for their assets. This for the client has solved the problem for them but not sorted the cause of the problem.