Welcome to WebmasterWorld Guest from 54.226.133.245

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Very high Traffic Analyzers

What are the top 100 sites using for log analysis?

     
12:40 am on Aug 18, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 11, 2003
posts:130
votes: 0



What do sites that get 30-50 million hits per day use to analyze log files? Sites like Amazon, Yahoo, AOL, or even high traffic adult sites?

What are the best log analyzers out there that can handle extemely high volume and still let you get detailed reports?

Is this something where you'd be better off importing log files into Oracle, or any high end RDMS, and have a high end DBA make custom reports? Or is there something out there that large sites use?

12:49 am on Aug 18, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 3, 2002
posts:1590
votes: 0


Most of them use their own custom systems to do their log analysis. When you get to that sort of size each business unit will have wildly different requirements and it just becomes easier in the long term to do something yourself.

I can't speak with 100% certaintly for all big sites of course but I know that one of the sites which you mentioned do their own custom log analysis.

11:12 pm on Aug 31, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 18, 2003
posts:827
votes: 0


Mozilla.org is using The Webalizer: [mozilla.org...]

The results for September through June were caused by log file truncation (the logs grew to larger than 2G).

3:13 pm on Sept 1, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 2, 2003
posts:95
votes: 0


Wow, moilla.org is really a mega site...

But whats with all the referrels from porn sites? Spamming I'm guessing...

3:15 pm on Sept 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 18, 2003
posts:827
votes: 0


xmlmania: Nope! Those sites all promote Firefox for "safe browsing". Check them out (from home, preferably).

They did have some mega-casino sites earlier, but they seem to have been flushed out now.

3:15 pm on Sept 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


tail & grep ;)
3:20 pm on Sept 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 8, 2004
posts:1679
votes: 0


tail & grep

rotate (more often than once a day) and bzip2 (x2 gain over gzip).