Best text editor for large log files

For Windows

[Macguru]

Hi,

I just want some input about what is the best text editor to edit very large log files.

I work on this site with a lot of traffic. Logfiles are huge. I need a program just to copy paste montly logs together. My windows PC got 1.5 gig of ram. Both Word and Open Office literally choke on the task.

What to use?

[Sinner_G]

How huge?

I had the problem with 1Gb files and ended up importing them into Access because no other application could handle that amount.

UltraEdit seemed to be able to handle the most of it.

[Woz]

You could try [url=http://www.notetab.com/]NoteTab[/url]. Not sure what the maximum size file it can handle but I have used it to do a search/replace on about 50 files at once so it can handle large amounts of data. It may do the job.

Onya
Woz

[Macguru]

>>How huge?

Average 1.8 gig a month in raw log format. They get 15 000 uniques a day.

[Sinner_G]

Average 1.8 gig a month in raw log format

Wow, I don't know of any text editor able to read that.

If you just want to copy/paste the files, wouldn't Norton Commander do the job?

[Macguru]

Sorry Woz, NoteTab is also choking on those chunks. Thanks I now have a good text editor for HTML on Windows.

>>Norton Commander

I will give it a try. Did not hink about it before. Off for a family BBQ.

(I will miss the quiz! :( )

[Pushycat]

As I recall, free NoteTab has a 16mb limit while the pro version has a 2gb limit. It usually helps if you check View/Options/Files/Filter Binary Codes.

[Woz]

"Grasshopper, when admiring the forest, remember to see the trees!"

I forgot about DOS! Try this, it should work.

Assuming: your log files are in c:\logs\files and are called something.log,

Shell out to DOS
cd c:\logs
copy files\*.log alllogs.log
exit (when finished)

That should concetenate all the files in the \files directory into one file in the \logs directory called, you gottit, alllogs.log

Should work, let us know.

Onya
Woz

[tedster]

The way I've been dealing with this challenge is to save the daily files as they are - and then zip up the bundle at the end of the month into one compressed file.

There's no need to concatenate that way, and it's easy to focus on specific time periods if I ever need to dig into the archives.

Neat trick with DOS, Woz! I forgot the little bit of DOS I once knew.

<added>
I also use EditPad for huge files - never noticed any limit (and they claim there is none). There's a free trial version, a shareware Lite version and reasonably priced Pro version. It's really a doozy of a text editor. Runs rings around many others.

Another really neat app for dealing with log files on Windows is Windows Grep. Someone here turned me onto it a few weeks ago, and it's changing my life (well, almost).
</added>

[Macguru]

Woz, that sounds good.

I will try it on Monday, when I can get acess to a Windows PC.

What I do now, is to run Analog on montly files. The output is set to a COMPUTER format (report.dat) I save it in a TAB format, import it in a FileMaker Pro database for crunching with older datas, export it again to tab to finally run Report Magic on the final report.dat file.

DOS sounds like the solution. Thanks Woz!

[clickclick]

I am told that an old copy of wordstar will handle extreemly large files limited only by hard disk space. Never tried it mind!

[Mark_A]

Macguru I recommend a product called textpad, there is no upper limit on the size of files you can load and editing options for selecting lines based on whatever criteria you want are excellent.

I am still on the free / share / demo but I like it so much I will almost certainly buy it. I think I got it from downloads.com

hth

[Macguru]

maccas came up with a very good solution for me in this tread. [webmasterworld.com...]

Text editors are not suited to render such files.

[Mark_A]

Interesting solution Macguru.

I needed a text editor for some log files because while they were large they were also corrupted with loads of hits from a windows server virus which added long long field entries. I wanted to do additional analysis in a custom database and the corrupted records spoiled the data import.

Textpad allowed the quick selection of just these corrupted lines from within an 100mb odd file and their deletion.

Additionally if you wanted to select and copy just entries for some IP, or requests for a element, for example to report illegal activity to a users ISP, textpad would allow you to do this in seconds from extremely large log files. I dont think the usual analysis packages give this kind of custom facility quite as easily.

[Macguru]

Maybe the thread title became less than acurate. I used text editors to copy paste "normal" logfiles together. My need for merging big files together did not need a text editor.

Of course, the program does not allow to edit them in any way. Some text editors maybe do it. The ones I tried just choke on it trying to open them.

I gess what we need now is a splitter program. To split logfiles in editable chunks.

My clients use different hosts. Some set the UNIX servers to have "clean" logfiles. Some are still pleaged with Windows worms scans.

I guess clean logs and email spam filters is becoming some important factor when some choose a web host or a ISP.

[Sinner_G]

To split logfiles in editable chunks

Hate to repeat myself ;), but again, Commander.

[Macguru]

OK.

But nowadays, I feel like some pain in the sacro illiac aerea reaching for my wallet and pulling the plastic from it. ;)

Any freebee around?

[martin]

I'm using EditPlus on Windows, it handles everything I throw at it but I'm not sure for 1.8G. A friend of mine told me he used UltraEdit to edit ODP's database ~800MB.

Hint: Use rar, ace, gzip or best bzip2 for compression rather than zip, the difference is really big. ON windows PowerArchiver supports bzip2 compression.

[martin]

Seeing all those Windows editors... does anybody know of a good Unix/Linux one that can handle very long lines (eg. 20-25K). None works for me.

[bcc1234]

Check out VIM.
It takes getting used to, but once you learn it - you'll love it.

[bird]

Mac, does it really have to be on Windows? Don't you have an OS-X box or linux sitting around?

If all else fails, have a look at cygwin [cygwin.com].
This will bring you the power of the standard unix tools to your Windows box.
Split files: split [hmug.org]
Concatenate files: cat [hmug.org]
Remove bad lines from a file: grep [hmug.org]

Edit files: gvim [vim.org] (available for Windows as well)

• Maximum line length On machines with 16-bit ints (Amiga and MS-DOS real mode): 32767, otherwise 2147483647 characters. Longer lines are split.
  
• Maximum number of lines 2147483647 lines.
  
• Maximum file size 2147483647 bytes (2 Gbyte) when a long integer is 32 bits. Much more for 64 bit longs. Also limited by available disk space for the swap-file.