Finding bandwidth hogs

[Mike521]

Our site routinely goes over our host's allotted monthly bandwidth, not that big of a deal but we pay extra for the additional bandwidth.

Is there a tool (either desktop or server-based) that can analyze a month's worth of log files and come up with a breakdown of the top files by bandwidth, along with referrer info so I can verify that the traffic is legit? Also useful would be user agent info so I can determine if there are useless bots wasting bandwidth too

Our host has a plesk stat log analysis tool which is handy, but it doesn't get specific enough (it tells me that jpg images take x amount of bandwidth, but not which images in particular or user agent / referrer info for those images)

Thanks in advance for any suggestions

[Staffa]

I would suggest that you start by comparing your raw log files with everything you can find in [webmasterworld.com...]

You will be amazed at how much bandwidth is wasted

[onlineleben]

There are a few 3rd party products around that even in their free version can give you a good indication of what is using up your bandwidth. Usually they used for optimizing load time, but I found that applying the recommendations also resulted in reducing bandwidth significantly. (I will send you a link to a tool via sticky - not affiliated but happy user of that tool).

[onlineleben]

If possible, try to switch on compression for HTML / CSS / JS files with GZip. Saves a lot!

[motorhaven]

In addition to compression, image cache headers are your friend!

[enigma1]

Maybe something like piwik web analytics will help.

[motorhaven]

Do you have shell access and raw Apache server logs? If so, I can give you some onliner awk commands which will do exactly what you need.

[Mike521]

I just noticed this thread in my control panel and that I never responded to anyone, I apologize, my email notifications go to an address that I almost never look at.

anyway I did do some log analysis and found a couple of things that were legit, so it does seem like it's ok

motorhaven if you still have those awk commands though I'd definitely give it a shot, we have about 2 months of logs on a rolling basis

thanks!