Welcome to WebmasterWorld Guest from 188.8.131.52
I just want some input about what is the best text editor to edit very large log files.
I work on this site with a lot of traffic. Logfiles are huge. I need a program just to copy paste montly logs together. My windows PC got 1.5 gig of ram. Both Word and Open Office literally choke on the task.
What to use?
I forgot about DOS! Try this, it should work.
Assuming: your log files are in c:\logs\files and are called something.log,
Shell out to DOS
copy files\*.log alllogs.log
exit (when finished)
That should concetenate all the files in the \files directory into one file in the \logs directory called, you gottit, alllogs.log
Should work, let us know.
There's no need to concatenate that way, and it's easy to focus on specific time periods if I ever need to dig into the archives.
Neat trick with DOS, Woz! I forgot the little bit of DOS I once knew.
I also use EditPad for huge files - never noticed any limit (and they claim there is none). There's a free trial version, a shareware Lite version and reasonably priced Pro version. It's really a doozy of a text editor. Runs rings around many others.
Another really neat app for dealing with log files on Windows is Windows Grep. Someone here turned me onto it a few weeks ago, and it's changing my life (well, almost).
I will try it on Monday, when I can get acess to a Windows PC.
What I do now, is to run Analog on montly files. The output is set to a COMPUTER format (report.dat) I save it in a TAB format, import it in a FileMaker Pro database for crunching with older datas, export it again to tab to finally run Report Magic on the final report.dat file.
DOS sounds like the solution. Thanks Woz!
I am still on the free / share / demo but I like it so much I will almost certainly buy it. I think I got it from downloads.com
I needed a text editor for some log files because while they were large they were also corrupted with loads of hits from a windows server virus which added long long field entries. I wanted to do additional analysis in a custom database and the corrupted records spoiled the data import.
Textpad allowed the quick selection of just these corrupted lines from within an 100mb odd file and their deletion.
Additionally if you wanted to select and copy just entries for some IP, or requests for a element, for example to report illegal activity to a users ISP, textpad would allow you to do this in seconds from extremely large log files. I dont think the usual analysis packages give this kind of custom facility quite as easily.
Of course, the program does not allow to edit them in any way. Some text editors maybe do it. The ones I tried just choke on it trying to open them.
I gess what we need now is a splitter program. To split logfiles in editable chunks.
My clients use different hosts. Some set the UNIX servers to have "clean" logfiles. Some are still pleaged with Windows worms scans.
I guess clean logs and email spam filters is becoming some important factor when some choose a web host or a ISP.
Hint: Use rar, ace, gzip or best bzip2 for compression rather than zip, the difference is really big. ON windows PowerArchiver supports bzip2 compression.
If all else fails, have a look at cygwin [cygwin.com].
This will bring you the power of the standard unix tools to your Windows box.
Split files: split [hmug.org]
Concatenate files: cat [hmug.org]
Remove bad lines from a file: grep [hmug.org]
Edit files: gvim [vim.org] (available for Windows as well)