Webtrends brought down my sites?

[askipper]

I am testing webtrends with their free trial and last night all my sites went down whilst it was building the database for Jan 05's log files.

When Webtrends finisehd the sites came back up

I didn't think webtrends could do this until having a chat with the guys from Netracker and they mentioned that Netracker could go and parse all the URL's to get the page titles.

That made me sit up - I noticed that Webtrends had the page titles of my pages in it too.

When I think back to last night I noticed that it was the databases that had gone down rather than the sites (though the loading times were horrific even for the html pages).

Could Webtrends have overloaded my database/s last night when it was loading and parsing the pages for the titles from the pages? Especially if it was hitting the database with database requests for all the database URL's.

If it did then "jeeze" it could have warned me! I get around 3000 UV's a day and on average they probably hit the database/s around 3-5 times each - so say 15,000 a day x 30 days = 450,000 calls to the database each month. Would webtrends have tried to do this itself over the 1.5 hours it was building the analytics database? If so, then I'm not surprised my sites went down

Any thoughts would be good on whether this could have been the cause and if so, can I stop it?

Thanks

[cgrantski]

There's a setup toggle called "retrieve HTML titles" or something like that, on the main setup page. If you turn that on, WT will retrieve the titles for the page URLs that are in your logs, but only once per URL. In other words, you could have 450,000 page views per month but WT only goes after the uniques, so the problem is really the # of unique pages, which could be, what, 1000? So that first analysis does hit your db pretty hard for a few minutes, then it starts to see repeats in the URLs and will only hit it again for the next unique URL that it does not already have in the cache it is building. I think WT ignores any session identifiers it finds in the URLs; I don't know if that applies to your site. You can look at the list it built by finding a folder called "datfiles" then opening the "titles" folder within it.

You can avoid the whole thing 2 ways: don't turn on "Retrieve HTML Titles," or you can supply WT with a "your site's domain" string that's either nonexistent or empty. But either way, once it has its cache of titles built, it won't bother your db again until the cache fills up or expires.

You can also see the WT activity by opening your logs; I think "WebTrends" will be in the User Agent field.

I agree that WT (and all such programs) should present a clear warning about this. In my experience, the load only lasts a very few minutes, but sometimes the sys admins get a little wigged out if they notice it going on. I haven't experienced it bringing down a database but that probably depends on configuration issues.

[askipper]

Thanks for all of that.

[zgb999]

Did you run Webtrends on the server where your website is running?

If your server is not strong enough this can be a problem too.

[askipper]

NO, it was run locally on a fairly low spec PC - it may all have been a coincidence to be honest but I shall make sure I run webtrends without titles in the future!

Thanks

[pmkpmk]

I don't know if it is of significance, but I just upgraded from "WT LogAnalyzer 8" to "WT SmallBusiness 7" (screw their Marketing Department and their version number system), and after I have migrated the profiles and it started to re-analyze the existing logfiles, my sites were down too (running on Apache on Linux). I restartet the httpd and the sites were up and running again.

WT was run on a dedicated PC (actually the PC where LogAnalyzer 8 resided was refused due to not meeting the specs - the new PC cost me 5x more than the upgrade of WT), while Apache run on a different machine. The logs were on the Apache machine. I use a CMS which relies on a MySQL database.

[pmkpmk]

It did it again!

The default for concurrent title requests was 100 (on a scale from 1-1000). I disabled page title requests now (I don't need them anyways).