High traffic stats service?

[chuckee]

Hi,
I have a forums website that I think gets more than 30 million pageviews a month. At any rate, I can't find any statistics services out there who are prepared to provide their service to me.
I was wondering if anyone knew of any stats/tracking service that would allow such a busy website to find out simple things such as
1) the number of page views
2) the number of unique visitors
3) referal URLs

I haven't been able to find any!
Thanks

[plumsauce]

Chuck,

I know of a service that has been heavily tested at 5,000 requests per second. Since there are 86,400 seconds in a day, the steady state capacity is about 430 million requests per day. Using a rule of thumb that your traffic is done in a 8 hour peak window, you need an equivalent capacity of 3 million requests per day. At those rates, the response time at the wire has been measured as being under 5 milliseconds for the 95th percentile.

That is,

30 million/month=1 million per day

being done in 8 hours of peak time, is also

1,000,000/8/60/60 = 34 requests/second

The service uses specialised purpose built software rather than depending on generic web server software to achieve its performance.

Since the TOS prohibits posting url's I will send you a private message with the info.

Plumsauce

++

[Receptional]

You'll be getting stickies now :)!

[plumsauce]

url drop cleanup please?

In the meantime, Stats, since a dedicated server is required, what is the upper limit without load balancing?

[cgrantski]

I'm surprised you haven't been able to find any, because 30 million a month is big but not by any means impossible. Is it that you haven't been able to find any at a price you are willing to pay?

[chris3ds]

Problem is not caused by the number of the requests but by the hard drive speed.

Eg: for a simple counter.

Load the counter file:
add 1 to the counter. (increase the counter)
Write and save the updated file

Now imagine, 5000 requests in the same times.

Counter=1
The counter File is loaded 5000 Times in the same times.
Each process increases the counter of 1. (Counter value=1) Add 1 (counter value=2)
Write and save the 5000 times the updated file

The counter value will be of 2 and not 5001.

So, you have to change the algo.

- Is the counter file is busy?
--> Yes
Please Wait until the file is free.
Load the counter file
block the file
increase the counter
write and save the counter.

--> No
Load the counter file
block the file
increase the counter
write and save the counter.

So during few ms you have 4999 processes running and checking the statut of the file.

For this reaseon, very few service accepts the high volume website. So this kind of accounts requests a dedicated server.

On a Dual Xeon 2.4Ghz 2xSCSI HD 15,000 RPM, 150 million PV per month is possible but it depends on the service provided.

Chris

[plumsauce]

Chris,

I will agree with you that tracking high page counts with a simplistic file based system is extremely difficult.

However, here:

For this reaseon, very few service accepts the high volume website. So this kind of accounts requests a dedicated server.

On a Dual Xeon 2.4Ghz 2xSCSI HD 15,000 RPM, 150 million PV per month is possible but it depends on the service provided.

I will have to respectfully disagree, except for the "very few services" part.

The number you are quoting works out to:

150 million pv/month

150/30=5 million pv/day

5 million/86400=57.87 pv/second

presuming steady state usage.

57/second is not even close to the 5000/sec that I described in the preceding post.

The problem with the 57/sec scenario is that it is at steady state over a 24 hour period. When the client site serves up 57+1 pages during a peak, traffic counts start to get dropped. Prior to that, the volume may already cause the pages to load slower because the traffic counter service is not responding fast enough under load.

The slow loading problem is seen frequently all over the web. The base page loads fast, but looks slow because it contains requests for images or ads on a third party server that is choking. The result is, the site looks slow through no fault of its own.

The takeaway here for all webmasters is to check on the responsiveness of all third party elements contained in their pages if they want the overall end user experience to be optimal.

If that number, 57/sec, already requires a dedicated server, then it is not scalable beyond that number short of load balancing amongst multiple boxes. This then opens the door to further difficulties because the client wants to see consolidated stats and not stats:part(a) and stats:part(b).

So, somewhere along the line, the stats have to be consolidated. Keep in mind that the counting is actually the easy part, after all both apache and iis can log at rates greater than 57/sec. The difficult part is all the behind the scenes manipulation that comprises the end product - web analytics.

Finally,

Is the 5000/sec possible? Yes.
Is it difficult? Yes.
Is it real? Yes.

Thanks for the chance to climb on my soapbox :)

Plumsauce

ps. mods: thanks for the url cleanup
+++