Webtrends visited each page 60 times in last weaks

So many hits from Webtrends - is this normal?

[zgb999]

For a site with 600 pages we had 36'000 hits from Webtrends since the beginning of the year.

Not even Googlebot is as actively crawling as Webtrends. What is Webtrends looking for so often?

[cgrantski]

Do you know it's WebTrends because of the IP or because it's in the UA field?

If it's in the UA field, do you know who's running it? It's not clear whether you're running WebTrends and don't understand why it should visit your site, or if some stranger is doing it without permission. Somebody using WebTrends to analyze your site's logs will cause hits to your site to retrieve page titles (because titles don't get into logs), but it doesn't do a crawl - it just goes to the URLs that it finds in the logs. If that's what's going on, WebTrends should be hitting each URL only once, though, and maybe doing it again in a few months when its page title cache file expires. So somebody needs to adjust some settings if each page gets hit 60 times since January 1.

On the other hand, there are some WebTrends-related alert & monitoring tools that will crawl a site looking for broken links or to ping pages to see if they're up, and so on. Or there used to be such tools. Anybody could be using those to get information about your site.

Those are the only things I can think of for the UA field.

[zgb999]

Hi Chris

Thank you for your answer!

The UA shows WebTrends/3.0 and my IP adress. So it must be my Webtrends installation bringing those hits.

The setting for "Number of days to keep cache" is 14. I guess this is the standard setting as I never changed it (and only found it as you mentioned the cache...). I could increase the number of days for the cache but even with 14 days in the cache I should never get as many hits.

[cgrantski]

By any chance, does your web site have a session or user identifier in the URL? In that case, every single URL in your logs will be completely unique (well, almost every one) so WT will hit your site approximately once for every page view, ever. If that's the case, turn off HTML Title Retrieval.

The program will also hit your server if the cache is full - the "cache" being the file full of page titles in the Titles folder under Datfiles in your configuration folder. The default is 50,000 for the current version of WT, so the only way your cache can be full is if your URLs are completely unique as described above. That brings us back to the hypothesis I started with - you've got something totally unique about your URLs.

Did I guess right?

[zgb999]

No, all URLs are static with no parameter or identifier at all (just widget-gadget.htm).

The files under datfiles/titles are not larger than 191 kb.

[cgrantski]

Hm. So much for the easy explanation. I'll have to do some research. Is it evenly spread out across days?

[zgb999]

There are peaks and some days with few hits. Below the stats for the hits from Webtrends for this year:
01/01 0
02/01 661
03/01 1.049
04/01 62
05/01 966
06/01 154
07/01 78
08/01 786
09/01 103
10/01 78
11/01 78
12/01 78
13/01 78
14/01 78
15/01 769
16/01 138
17/01 78
18/01 1.340
19/01 1.578
20/01 2.323
21/01 760
22/01 1.355
23/01 1.222
24/01 955
25/01 86
26/01 86
27/01 1.984
28/01 192
29/01 93
30/01 86
31/01 163
01/02 2.259
02/02 151
03/02 87
04/02 87
05/02 87
06/02 4.633
07/02 1.808
08/02 737
09/02 111
10/02 117
11/02 1.524
12/02 277
13/02 106
14/02 51
15/02 87
16/02 87
17/02 0
18/02 0
19/02 87
20/02 5.489
21/02 2.467
22/02 1.016

Usualy the logfiles are only downloaded once a week on Sunday sometimes I manually download them before that. Maybe the peaks are related to that.

[cgrantski]

That's really, really strange. On what days did you run reports?

[zgb999]

Maybe it was on the days with peaks but I cannot say for sure.

[cgrantski]

yeah but if you're not running an analysis on other days, and there are still hits ....

[zgb999]

I found out that when I am not downloading new logfiles then there is (almost) no traffic with the Webtrends useragent.

As soon as I download logfiles and those logfiles get analysed I have thousands of hits (though the site only has about 600 pages). Yesterday I had 7400 hits from Webtrends only because it had to analyze the latest data from newly downloaded logfiles.

In the options the setting for the cache of titles was 14 and I changed it to 60 now to see what change this causes.

[sagecreek]

any updates on what the change had on your activity if any? I'm thinking of buying webtrends...

Thanks! -CW

[zgb999]

I will probably be ready on Monday to tell you more.

[zgb999]

As mentioned I found that when I am not downloading new logfiles then there is (almost) no traffic with the Webtrends useragent. So I did not download new logfiles and got the following hits after downloading the logfiles after a week without download:
week 1 with 14 days cache: about 12 hits per page of the site for analyzing the logfiles for the week.
week 2 with 60 days cache: about 5 hits per page of the site for analyzing the logfiles for the week.

So the cache setting seems to influence the amount of hits but still I don't see why with a cache of 60 days each page should be hit 5 times.

[cgrantski]

I am even more sure this has something to do with page titles, because your research is pointing toward it. What version of WT is this? (the whole name plus number ... could be WebTrends Log Analyzer 8.1? Small Business 7.0b? etc? Also, please check the available disk space where you keep the program.

[zgb999]

We use WebTrends Reporting Center - Enterprise Edition 6.1 (Build Number: 7538)

Disk space: 54 GB available.