I work for a company with a highly trafficked site (about 100,000 unique daily visitors). We have our "MANAGED" hosting through Rackspace for quite the pretty penny per month. It includes a dedicated back end / DB server and cloud servers for the front end.
The setup has been functioning perfect for 10 months now; but this past Monday the speed of the site immediately dropped. Page load times fell from 1-2 seconds to between 10-20 seconds, and sometimes not at all. As far as we know (and as far as Rackspace says), no server setting were modified. No new code was introduced on our end. It's a mainly static site, with minimal user interaction with the backend at all.
Can any expert offer some advice? We've monitored the traffic, checked IPs, etc. We've even tunes down several site features in the interest of reducing server load. Upon a server reboot, the active threads/processes running on it IMMEDIATELY jump back up to maxed out levels. It seems like once our daily traffic reaches 10MB/s, a type of queue forms and the delays begin. Rackspace assures us that we're not limited to that.
Please advise - thanks! -Jay-
EDIT: Some more background info: The site is typically busiest from 7am until 3pm EST. For the past few days, we've noticed that between 7pm and 9-10pm the server has just lagged incredibly. However, at around that 9-10pm mark, something changes and the pages go back to loading almost instantly. (There is still decent traffic though.) Then at around 7am again it slows to a crawl.
Rackspace has offered solutions such as spinning up another server and incorporating their load balancing - they are in the process of this BUT they do NOT think the traffic is the issue. At one point they actually said there was potential packet loss somewhere in the network, but no progress has been made.
Versions in Use:
OS: cent OS on cloud
OS: Redhat on Dedicated Server
PHP: 5.3 / MySQL: 5.1.69