Tracking Big network of sites

[protheus99]

What's the best way to track a major network of sites in real time? We stopped using log's and have databased most of the information. But the number of our sites makes it almost impossible to do 100 different sql query to check out these different sites. Is there a better way for a network of 250 that is constantly growing and will soon have 1000?

[polythene]

Just some thoughts: Is it possible to consolidate the logging to one database? To make it real time it would help to write it all to the same place, which wouldn't be difficult if all sites are hosted on the same network as the DB. Otherwise accept some lag time and set up jobs to periodically consolidate stats.

Of course the number of queries depend on what data you want out of it, but for example if it's all in one table you could have a field to identify the hostname of each site. Then you could group by that field to get things like sum of hits for each site, etc.

[netguy]

protheus99, in-depth analysis of 100 or more sites is a difficult challenge, but if you just want to monitor real time impressions only - and your total traffic is under 10,000/hr - then you might want to consider using an ad management database.

In your ad management software (on one of your servers), you just set up different accounts like you were placing a banner on separate websites, and just use a 1px transparent gif at the bottom in a JS snippet for each site.

The end result is you have a single ad/site management page online with all your websites and select each domain name to find immediately how many impressions you are getting (per hour/day/month) in real time.

This can be very helpful in quickly determining what is happening (or not happening) on each site - then grab the full logs to determine the details.

This worked great for several years. In 10 seconds we could select a different site and immediately see what impressions an hour we were getting (across several dozen sites).

Unfortunately (or fortunately), we were hitting more than 10,000 impressions/hr too often on the sites we were monitoring, and it would blow out the MySQL database.

Now on the larger sites, we just compress the logs (gz) daily to save about 90% of the download file size, then run them individually through WebTrends.

So if your 'major network' is refering more to the number of sites, than traffic, the ad management route may work for you.

Otherwise, hopefully this thread stays open long enough to where we can get some other input for large network 'and' large traffic, real-time solutions.

Steve