homepage Welcome to WebmasterWorld Guest from 54.163.72.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Best text editor for large log files
For Windows
Macguru




msg:891994
 2:11 pm on Aug 3, 2002 (gmt 0)

Hi,

I just want some input about what is the best text editor to edit very large log files.

I work on this site with a lot of traffic. Logfiles are huge. I need a program just to copy paste montly logs together. My windows PC got 1.5 gig of ram. Both Word and Open Office literally choke on the task.

What to use?

 

Sinner_G




msg:891995
 2:18 pm on Aug 3, 2002 (gmt 0)

How huge?

I had the problem with 1Gb files and ended up importing them into Access because no other application could handle that amount.

UltraEdit seemed to be able to handle the most of it.

Woz




msg:891996
 2:21 pm on Aug 3, 2002 (gmt 0)
You could try [url=http://www.notetab.com/]NoteTab[/url]. Not sure what the maximum size file it can handle but I have used it to do a search/replace on about 50 files at once so it can handle large amounts of data. It may do the job.

Onya
Woz

Macguru




msg:891997
 2:30 pm on Aug 3, 2002 (gmt 0)

>>How huge?

Average 1.8 gig a month in raw log format. They get 15 000 uniques a day.

Sinner_G




msg:891998
 2:37 pm on Aug 3, 2002 (gmt 0)

Average 1.8 gig a month in raw log format

Wow, I don't know of any text editor able to read that.

If you just want to copy/paste the files, wouldn't Norton Commander do the job?

Macguru




msg:891999
 2:53 pm on Aug 3, 2002 (gmt 0)

Sorry Woz, NoteTab is also choking on those chunks. Thanks I now have a good text editor for HTML on Windows.

>>Norton Commander

I will give it a try. Did not hink about it before. Off for a family BBQ.

(I will miss the quiz! :( )

Pushycat




msg:892000
 3:02 pm on Aug 3, 2002 (gmt 0)

As I recall, free NoteTab has a 16mb limit while the pro version has a 2gb limit. It usually helps if you check View/Options/Files/Filter Binary Codes.

Woz




msg:892001
 10:18 am on Aug 4, 2002 (gmt 0)

"Grasshopper, when admiring the forest, remember to see the trees!"

I forgot about DOS! Try this, it should work.

Assuming: your log files are in c:\logs\files and are called something.log,

Shell out to DOS
cd c:\logs
copy files\*.log alllogs.log
exit (when finished)

That should concetenate all the files in the \files directory into one file in the \logs directory called, you gottit, alllogs.log

Should work, let us know.

Onya
Woz

tedster




msg:892002
 11:47 am on Aug 4, 2002 (gmt 0)

The way I've been dealing with this challenge is to save the daily files as they are - and then zip up the bundle at the end of the month into one compressed file.

There's no need to concatenate that way, and it's easy to focus on specific time periods if I ever need to dig into the archives.

Neat trick with DOS, Woz! I forgot the little bit of DOS I once knew.

<added>
I also use EditPad for huge files - never noticed any limit (and they claim there is none). There's a free trial version, a shareware Lite version and reasonably priced Pro version. It's really a doozy of a text editor. Runs rings around many others.

Another really neat app for dealing with log files on Windows is Windows Grep. Someone here turned me onto it a few weeks ago, and it's changing my life (well, almost).
</added>

Macguru




msg:892003
 12:58 pm on Aug 4, 2002 (gmt 0)

Woz, that sounds good.

I will try it on Monday, when I can get acess to a Windows PC.

What I do now, is to run Analog on montly files. The output is set to a COMPUTER format (report.dat) I save it in a TAB format, import it in a FileMaker Pro database for crunching with older datas, export it again to tab to finally run Report Magic on the final report.dat file.

DOS sounds like the solution. Thanks Woz!

clickclick




msg:892004
 11:37 pm on Aug 4, 2002 (gmt 0)

I am told that an old copy of wordstar will handle extreemly large files limited only by hard disk space. Never tried it mind!

Mark_A




msg:892005
 10:44 am on Aug 5, 2002 (gmt 0)

Macguru I recommend a product called textpad, there is no upper limit on the size of files you can load and editing options for selecting lines based on whatever criteria you want are excellent.

I am still on the free / share / demo but I like it so much I will almost certainly buy it. I think I got it from downloads.com

hth

Macguru




msg:892006
 3:09 pm on Aug 7, 2002 (gmt 0)

maccas came up with a very good solution for me in this tread. [webmasterworld.com...]

Text editors are not suited to render such files.

Mark_A




msg:892007
 6:10 am on Aug 8, 2002 (gmt 0)

Interesting solution Macguru.

I needed a text editor for some log files because while they were large they were also corrupted with loads of hits from a windows server virus which added long long field entries. I wanted to do additional analysis in a custom database and the corrupted records spoiled the data import.

Textpad allowed the quick selection of just these corrupted lines from within an 100mb odd file and their deletion.

Additionally if you wanted to select and copy just entries for some IP, or requests for a element, for example to report illegal activity to a users ISP, textpad would allow you to do this in seconds from extremely large log files. I dont think the usual analysis packages give this kind of custom facility quite as easily.

Macguru




msg:892008
 4:50 pm on Aug 8, 2002 (gmt 0)

Maybe the thread title became less than acurate. I used text editors to copy paste "normal" logfiles together. My need for merging big files together did not need a text editor.

Of course, the program does not allow to edit them in any way. Some text editors maybe do it. The ones I tried just choke on it trying to open them.

I gess what we need now is a splitter program. To split logfiles in editable chunks.

My clients use different hosts. Some set the UNIX servers to have "clean" logfiles. Some are still pleaged with Windows worms scans.

I guess clean logs and email spam filters is becoming some important factor when some choose a web host or a ISP.

Sinner_G




msg:892009
 4:51 pm on Aug 8, 2002 (gmt 0)

To split logfiles in editable chunks

Hate to repeat myself ;), but again, Commander.

Macguru




msg:892010
 5:02 pm on Aug 8, 2002 (gmt 0)

OK.

But nowadays, I feel like some pain in the sacro illiac aerea reaching for my wallet and pulling the plastic from it. ;)

Any freebee around?

martin




msg:892011
 12:32 am on Aug 9, 2002 (gmt 0)

I'm using EditPlus on Windows, it handles everything I throw at it but I'm not sure for 1.8G. A friend of mine told me he used UltraEdit to edit ODP's database ~800MB.

Hint: Use rar, ace, gzip or best bzip2 for compression rather than zip, the difference is really big. ON windows PowerArchiver supports bzip2 compression.

martin




msg:892012
 12:48 am on Aug 10, 2002 (gmt 0)

Seeing all those Windows editors... does anybody know of a good Unix/Linux one that can handle very long lines (eg. 20-25K). None works for me.

bcc1234




msg:892013
 9:14 pm on Aug 11, 2002 (gmt 0)

Check out VIM.
It takes getting used to, but once you learn it - you'll love it.

bird




msg:892014
 9:43 pm on Aug 11, 2002 (gmt 0)

Mac, does it really have to be on Windows? Don't you have an OS-X box or linux sitting around?

If all else fails, have a look at cygwin [cygwin.com].
This will bring you the power of the standard unix tools to your Windows box.
Split files: split [hmug.org]
Concatenate files: cat [hmug.org]
Remove bad lines from a file: grep [hmug.org]

Edit files: gvim [vim.org] (available for Windows as well)

  • Maximum line length On machines with 16-bit ints (Amiga and MS-DOS real mode): 32767, otherwise 2147483647 characters. Longer lines are split.
  • Maximum number of lines 2147483647 lines.
  • Maximum file size 2147483647 bytes (2 Gbyte) when a long integer is 32 bits. Much more for 64 bit longs. Also limited by available disk space for the swap-file.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved