Easy way to chrono-sort logs to enable Webtrends?

Webttrends chokes on out-of-sequence logs

[paybacksa]

I need a way to sort apache logs into chrono order so that WebTrends will process them, and I only have Windows available. Any ideas?

The web host apparently updates server time every few days, and the logs are filled with what WeBtrends considers out-of-time-sequence entries, and it won't process the logs. Available Windows apps don't behave well with hundreds or thouands of lines (notepad, Write, Excel, etc) and I would prefer NOT to write my own Access import/export routines for this.... unless necessary.

Anyone know of a VIM script readly available?

[cgrantski]

I use TextPad or Ultra Edit to do this when needed; they have no size limit.

[flashfan]

And UltraEdit can sort by columns.

[john_k]

How about DOS SORT

SORT < filename.log > newfilename.log

I'm not sure about any filesize limits.

[cgrantski]

I tried DOS SORT - definitely does have filesize limits.

And I want to add the of the two I mentioned, Ultra Edit with its ability to sort by columns is far superior because the date and time aren't always at the beginning of the line. I forgot about that aspect.

[paybacksa]

Thanks. Can you tell me is UltraEdit crippleware/leaseware/ etc? I seeit is free trial shareware - details appreciated.

[cgrantski]

If you don't care about upgrades, there's no additional cost after the purchase. There's an upgrade fee, except for the first year when upgrades are free. After the first year, you can wait for as long as you want and buy upgrades at any point for half the retail price, or you can buy a maintenance contract every year for about 20% of the retail price that gives you free upgrades during that year. I think there's a deal on "upgrades for life" also.

[john_k]

fyi - I just tried using DOS SORT on a 53meg log file and it sorted with no problems. It took about 30 seconds. This is on a Windows 2000 machine. The command also takes several parameters, one of which lets you specify the first character column on which to base the sort.

[paybacksa]

TextPad - will sort on the nth character in. Not useful since logs are not space delimited (column for date field varies with length of IP number)

DOS SORT - also limited to sorting on nth character of row. Not useful here.

UltraEdit - will sort, but first requires you to parse the file into columns. Looks like that is a major task, since it's been running for 15 minutes already with tons of disk activity.. I suspct it wil be parsed incorrectly but we shall see.

I have to guess that no one else has this problem?

[john_k]

I have to guess that no one else has this problem?

I've had to combine logs from load-balanced servers for web trends in the past. I think this may have been to get around the same issue you are having (it was two years ago at least). I used a VB program to do it though. It read a line from each of the source log files, determined which was earliest, wrote it to the output file, and then fetched another row from the same source file.

[chewy]

I've seen it, and floated a query to this forum not that long ago. I'm pleased to see more and different approaches emerging.

I've been doing it by hand (with TextPad)- so UltraEdit sounds pretty good to me.

What I've been seeing is both out of order and sets of duplicates, also out of order - so for me, I would want sort and de-dupe capabilities.