Forum Moderators: DixonJones
was wondering if any of you can help me to understand WebTrends.
1. The webservers that WebTrends will be accessing have the FTP ports turned off for security reasons. Is there any other method/way that WebTrends can get access to the logfiles on those webservers? (Filesharing is not an option either). One of the server is a Domino Server, which i think we can just use FTP to transfer the logfile, but the other two webservers are not Domino based (again, for security reasons, FTP is not allowed).
2. From your experience with other customers (mid to large companies), what are the best practice for the data rentention process? backup/storage process?
3. What are the different types of client access methods for the reports?
4. What is FastTrends? (I've done a google on this and nothing helped! :( )
5. When Webtrends collect the log file from the webservers, what is the size (rough mulipling factor) of the 'massaged' data that WebTrends would use for its reporting? for example, if the logfile is about 100MB in size, then when WebTrends get that logfile and processes that logfile for its own use for generating reports, what is the size of the file?
Any help is appreciated.. thanks in advance!
RR
Personally I use webtrends on it's own dedicated box, just found it easier and more stable. Every night after 1am (be careful with Daylight savings time) I have a script that FTP's logs from the webserver to the webtrends box. The webtrends analysis is scheduled to kick off shortly after that. This way my reports are all ready for me to view when I get in.
I burn all of the logs to CD every month and keep 1 year of files on the box. If you create new reports they can take a while to run so develop them using a 1 week time frame, then adjust the time period once you have the report configured correctly.
I use webtrends on an intranet (no external access) but I limit who can see what with different profiles.
FastTrends allows it to rerun the reports faster. However if you change a filter it has to reread the data. Thats why when developing a new report I use the 1 week timeframe mentioned above.
The size of logs depends on traffic and log configuration. Webtrends can read into zipped files as long as there is enough disk space to unzip it. Don't archive too much or too frequently, as it consumes lots of disk space (personally I archive once every 2 weeks and no more than 4 archives).
Anyhow, thanks for your quick response.
Just a few more things:
1. If there a size limitation to FastTrends storage? (like how MS Access has a limitation).
2. I'll be using WebTrends interally too, but is there any other way to get the logfiles without using FTP?
3. We were thinking of a 1 year rolling data retention... any ideas how to implement that?
Thanks again!
2. FTP was the easiset, you could write a VB Script to map a drive and then destroy the connection when it's finished. If you run the script on the webserver you really shouldn't have a problem.
3. I wrote a vbscript to delete the files based on the name of the file (20040318_ex.log) that runs once a week.
For example, a week's worth of Page Views is simply the sum of 7 individual days of page views; no need to count them all over again using the logs. But for stats like "unique visitors," WebTrends has to process the whole week as one block - because Visitor X may have visited once on Thursday, and once on Friday - but for accuracy, Visitor X needs to count as one, not two, Unique Visitors in the week-long report.
In other words, when doing a weekly report after doing a series of dailies, WebTrends uses FastTrends data and calculates sums of dailies for a lot of the stats (but not all), and creates a few stats directly from the logs. Saves a lot of processing.
But as GrayWolf says, if you change one of the premises of the reports (such as changing a filter), the FastTrends stored statistics will be based on the old filter. So, you have to re-process FastTrends, which as I recall involves first deleting the affected days using the FastTrends admin. If you don't stay on top of those changes, your reports will still run but they'll be wrong. If you don't do the deletions for example, FastTrends just appends the re-processed information to what it already has for the days affected, and you get funky doubled numbers.
The FastTrends database takes up quite a bit of room. You have to decide what periods of time can be deleted and stay on top of it.
These shortcomings may be corrected in recent versions of WebTrends Log Analyzer; my last experience was with version 5 or 6 of Analysis Suite.
I hope this long-winded explanation helps you avoid some of the mistakes I made.
I have more questions:
1a. I've read somewhere that FastTrends will process the logfiles and the summary file that it creates is about 5-25% of the size of the logfile, but somewhere else, it says it'll be 5-25% smaller than the log file. So which is it? is it 5-25% OF the logfile size, or 5-25% SMALLER than logfile? (that's a big difference).
1b. Typically, how large are the reports that WebTrends generate for each profile? (I suppose it's dependant on what info is requested)
2. The company I'm setting up WebTrends for is a large company. They only have 2 webservers that they want to analyze on, and one other Domino/Websphere server (used for remote email access)... the two webservers are load balanced, and they may have plans to increase the number of servers for WebTrends.
Each webserver generates about 60MB worth of logfile (thus a total of 120MB) daily!
Now, with WebTrends, are you saying that it will FTP/HTTP the logfiles to the FastTrends database, and then generate a report from there? Which I think is what I got from what you said above... if that is the case, then you would still need to archive/store the logfiles on the WebTrends server in the case the report profile changes (filter is modified). Is this correct?
3. I was thinking that for my client's WebTrends hardware recommendation they would have 4 hard Drives (10GB - running the OS and stuff, 80GB - that keeps the current rolling of 1 year, 200GB - for storing raw logfiles from the webservers, and 160GB - for FastTrends)... Just to get an idea of how other's typical configuration on the WebTrends server, what is yours?
Thanks again for your help!
Which product, and which version, of WT are you going to be using? That helps with estimating the size of the reports themselves.
Consider storing the logs on the servers themselves in their original location. As long as they're somewhere when you need them .... on the WT machine itself, you just need logs that go back a day further than the beginning of the longest report period. If monthlies will be your longest report period, then they need to go back to (and include) the last day of the previous calendar month. If you someday need to do something massive retrospectively, you can jury-rig something like a temporary extra hard disk.
I know I'm making things even more complicated!
As well based on what I've been told, a daily logfile of 100MB in size, WebTrends will analyze and summarize the data to 30MB in size (sounds right?)... and that's PER Profile (right again?)
Thus if I have about 30 Profiles running, that will be about 0.9 GB of data space that is taken up? How much does archiving take up if i do a weekly archiving?
Geez, I'm getting moreconfused by the second! :(
If your log files are only 120 MB in size per day, even a year's worth of uncompressed logs on your WT box would be less than 50 GB. There's an old rule of thumb (from WRC 4x) about 1.5 MB per report - retrospective reports for the last year for a total of 30 profiles and all dailies, weeklies, etc for each would be only 20 GB in all. The program itself wants a gig of space to work in. That works out to 50+20+1 GB for a year, compressing and archiving nothing.
If I were you I'd get a 200 GB hard drive (160's are $60 on sale at Circuit City) and not worry about disk space any more. Spend a little more on 512 MB RAM sticks and max it out and spend your time henceforth on reading the admin guide, scripting manual, and the rest.
I just returned from a 1 week trip.
Anyways, thanks for the feedback again... here's more questions:
Our client is using a web server specifically for users to access their emails remotely.. so the questions are: 1. can WebTrends report on how many incoming and outgoing emails thru this server? 2. can a report of how many users are on at any given time?
I guess the answers would be yes IF the logfiles are collecting these infos right? Does the log files collect these infos? if so, then i suppose WebTrends can generate a report on this right? If not, then how can we set up logfiles to collect these data?
RR
