|The Toughest Problem We've Ever Dealt With|
Transfer speed throttling for just Comcast Internet Users
My company runs an enterprise-level dedicated server with an extremely reputable managed hosting company whose staff have thoroughly impressed me with their skill and knowledge. But we are having an issue with our server that we just cannot figure out for the life of us, and I am coming to you in hopes that you may have an idea what the culprit is and how to rectify it. We are running CentOS/Apache2.2.
The problem we are having is that this particular server has very slow (150 kb/sec) transfer speeds through both browser and FTP for all Comcast Internet users (other ISP's, verizon, etc. are getting very fast transfer speeds). What's odd is all the other servers hosted at the datacenters of my hosting company are very fast for these same Comcast users who get very slow transfer speeds to my server. We looked into many things in order to find an answer to this problem, but were unable to find anything wrong with the server's configuration. We also exhaustively looked at traceroutes and how Comcast was routing traffic for it's internet users to my server in comparison to how it was routing traffic to the other nearly identical "fast" servers in the same datacenter as mine. We found nothing that looked suspicious.
So in an attempt to correct the situation, we migrated to a newly built server on a new subnet in hopes it would rectify the situation. It "sort of" did, but didn't... let me explain.
When we migrated to the new server, the transfer speeds Comcast internet users were getting were very fast at first, but about 20 hours later after putting the new server live, those Comcast users were getting the very slow 150 kb/sec transfer speeds to the new server, and it stayed that way.
But we found a temporary fix for this problem....
If we reboot the server, it fixes the speed issues for Comcast users and they are able to download at very fast speeds (over 1 mb/sec generally), but it only lasts for about 12-20 hours before everyone on Comcast starts seeing slow transfers again. If we then reboot the server when the slow speeds come back, it fixes the transfer speeds once again for about 12-20 hours before they fall back to being slow again. We can repeat this process of rebooting to fix the problem, but it only lasts a short period of time before the transfers go slow again (12-20 hrs).
And just to be clear, this server is no where near being overloaded. The server is far more powerful than the few websites that are on it need it to be. So we can assuredly say overloading isn't the problem. There is also no "questionable" content on any accounts that would be cause for concern as to banning/blocking by Comcast or anyone else.
We are at a complete loss as to what could be causing such a problem with this server. The worse thing is we really don't know if this is our problem, or Comcast's problem. But my gut says something may be awry with my server since I can at least temporarily rectify the transfer speeds problem with a reboot.
So I come to all of you in hopes that perhaps you could share some insight or ideas as to what may be going wrong here, and what might be able to be done to fix this. Thank you so much for your help. I truly appreciate it.
First of all a warm welcome to WebmasterWorld!
This is not really the type of post we expect here of a first time poster, but it is really appreciated :)
Makes me think of a firewall issue. You can program the iptables firewall under Linux in such a way that high speed bursts are allowed, but as soon as people start downloading large amount of data, the number of packets per second is limited. After the server is restarted the counters of the firewall are reset to zero, and high speeds are allowed initially.
You can look at your current rules in the iptables firewall with the following command (with root permissions):
iptables -n -L -v
If there are any IPs or IP ranges in the table that match IPs in the route to Comcast, that might be the reason of your problem.
Also look at netstat -nt when the problem happens to see if the connection states for the comcast users are different from that of other users.
Search for "Comcast" and "throttling". I assume you've heard of the allegations that Comcast is throttling Bittorrent traffic, and perhaps other traffic (most likely video, etc.)
Maybe you've been caught in an overly-wide net.
Since Comcast repeatedly denys the charges, it's possible that they have technical issues themselves.
More likely, though, caught in an overly-wide net.
Hey guys... thanks for your replies. We tried turning off iptables to see if that was causing any problems, Comcast users still were getting slow transfers. netstat's aren't showing any discrepancies, at least from my observations. I will have the admins take a look to see if they notice anything different. But from my view, everything looks okay.
I have been reading many things about Comcast's supposed, "Sandvine", which throttles bittorrent traffic. However, it seems that it only affects bittorrents, and there is certainly nothing on our accounts that even come close to torrents. But like you said, an overly-wide net could be to blame. Though we have tried three different IP ranges all in very different subnets. I will be continuing my research on this possibility for sure.
Have you examined this traffic in detail using a TCP/IP data logger?
If your host is not overly-burdened, you can run a data logger on your host. Otherwise, you can run it externally (hardware device or software). In that case, you will need a managed switch, and will have to set it to duplicate traffic from you server's port onto the logger's port.
I've had comcast throttle/cut off large downloads before - in my case backup files I was downloading from my server. So it's possible that it's not something you have any control over. Is there some way to test with the comcast users by asking them to download something off other servers to try to isolate it?
|I've had comcast throttle/cut off large downloads before - in my case backup files I was downloading from my server. |
FWIW, in contrast, Cox does the opposite.
They have a feature called PowerBoost that temporarily increases the total bandwidth available to the customer (beyond their contracted rate) when they are doing a large file download.
Each Powerboost only lasts a few seconds - so it doesn't increase bandwidth available for streaming media, etc.
But it speeds delivery of typical file downloads.
Note that bandwidth-shaping is used across the industry by almost all ISPs. Comcast is noted, though, for doing it in strange, incompatible ways that are unfriendly to client software.
I'd bet that your problem is at the Comcast end of things. I'd first make sure what service level your complaining customers have. I don't know anything about Comcast service tiers (I have Cox), but they might have a "budget" tier, similar to Cox's low-end offering, which offers very limited bandwidth.