homepage Welcome to WebmasterWorld Guest from 54.167.174.90
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Best text editor for large log files
For Windows
Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 2:11 pm on Aug 3, 2002 (gmt 0)

Hi,

I just want some input about what is the best text editor to edit very large log files.

I work on this site with a lot of traffic. Logfiles are huge. I need a program just to copy paste montly logs together. My windows PC got 1.5 gig of ram. Both Word and Open Office literally choke on the task.

What to use?

 

Sinner_G

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 565 posted 2:18 pm on Aug 3, 2002 (gmt 0)

How huge?

I had the problem with 1Gb files and ended up importing them into Access because no other application could handle that amount.

UltraEdit seemed to be able to handle the most of it.

Woz

WebmasterWorld Senior Member woz us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 2:21 pm on Aug 3, 2002 (gmt 0)
You could try [url=http://www.notetab.com/]NoteTab[/url]. Not sure what the maximum size file it can handle but I have used it to do a search/replace on about 50 files at once so it can handle large amounts of data. It may do the job.

Onya
Woz

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 2:30 pm on Aug 3, 2002 (gmt 0)

>>How huge?

Average 1.8 gig a month in raw log format. They get 15 000 uniques a day.

Sinner_G

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 565 posted 2:37 pm on Aug 3, 2002 (gmt 0)

Average 1.8 gig a month in raw log format

Wow, I don't know of any text editor able to read that.

If you just want to copy/paste the files, wouldn't Norton Commander do the job?

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 2:53 pm on Aug 3, 2002 (gmt 0)

Sorry Woz, NoteTab is also choking on those chunks. Thanks I now have a good text editor for HTML on Windows.

>>Norton Commander

I will give it a try. Did not hink about it before. Off for a family BBQ.

(I will miss the quiz! :( )

Pushycat

10+ Year Member



 
Msg#: 565 posted 3:02 pm on Aug 3, 2002 (gmt 0)

As I recall, free NoteTab has a 16mb limit while the pro version has a 2gb limit. It usually helps if you check View/Options/Files/Filter Binary Codes.

Woz

WebmasterWorld Senior Member woz us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 10:18 am on Aug 4, 2002 (gmt 0)

"Grasshopper, when admiring the forest, remember to see the trees!"

I forgot about DOS! Try this, it should work.

Assuming: your log files are in c:\logs\files and are called something.log,

Shell out to DOS
cd c:\logs
copy files\*.log alllogs.log
exit (when finished)

That should concetenate all the files in the \files directory into one file in the \logs directory called, you gottit, alllogs.log

Should work, let us know.

Onya
Woz

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 11:47 am on Aug 4, 2002 (gmt 0)

The way I've been dealing with this challenge is to save the daily files as they are - and then zip up the bundle at the end of the month into one compressed file.

There's no need to concatenate that way, and it's easy to focus on specific time periods if I ever need to dig into the archives.

Neat trick with DOS, Woz! I forgot the little bit of DOS I once knew.

<added>
I also use EditPad for huge files - never noticed any limit (and they claim there is none). There's a free trial version, a shareware Lite version and reasonably priced Pro version. It's really a doozy of a text editor. Runs rings around many others.

Another really neat app for dealing with log files on Windows is Windows Grep. Someone here turned me onto it a few weeks ago, and it's changing my life (well, almost).
</added>

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 12:58 pm on Aug 4, 2002 (gmt 0)

Woz, that sounds good.

I will try it on Monday, when I can get acess to a Windows PC.

What I do now, is to run Analog on montly files. The output is set to a COMPUTER format (report.dat) I save it in a TAB format, import it in a FileMaker Pro database for crunching with older datas, export it again to tab to finally run Report Magic on the final report.dat file.

DOS sounds like the solution. Thanks Woz!

clickclick

10+ Year Member



 
Msg#: 565 posted 11:37 pm on Aug 4, 2002 (gmt 0)

I am told that an old copy of wordstar will handle extreemly large files limited only by hard disk space. Never tried it mind!

Mark_A

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 565 posted 10:44 am on Aug 5, 2002 (gmt 0)

Macguru I recommend a product called textpad, there is no upper limit on the size of files you can load and editing options for selecting lines based on whatever criteria you want are excellent.

I am still on the free / share / demo but I like it so much I will almost certainly buy it. I think I got it from downloads.com

hth

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 3:09 pm on Aug 7, 2002 (gmt 0)

maccas came up with a very good solution for me in this tread. [webmasterworld.com...]

Text editors are not suited to render such files.

Mark_A

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 565 posted 6:10 am on Aug 8, 2002 (gmt 0)

Interesting solution Macguru.

I needed a text editor for some log files because while they were large they were also corrupted with loads of hits from a windows server virus which added long long field entries. I wanted to do additional analysis in a custom database and the corrupted records spoiled the data import.

Textpad allowed the quick selection of just these corrupted lines from within an 100mb odd file and their deletion.

Additionally if you wanted to select and copy just entries for some IP, or requests for a element, for example to report illegal activity to a users ISP, textpad would allow you to do this in seconds from extremely large log files. I dont think the usual analysis packages give this kind of custom facility quite as easily.

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 4:50 pm on Aug 8, 2002 (gmt 0)

Maybe the thread title became less than acurate. I used text editors to copy paste "normal" logfiles together. My need for merging big files together did not need a text editor.

Of course, the program does not allow to edit them in any way. Some text editors maybe do it. The ones I tried just choke on it trying to open them.

I gess what we need now is a splitter program. To split logfiles in editable chunks.

My clients use different hosts. Some set the UNIX servers to have "clean" logfiles. Some are still pleaged with Windows worms scans.

I guess clean logs and email spam filters is becoming some important factor when some choose a web host or a ISP.

Sinner_G

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 565 posted 4:51 pm on Aug 8, 2002 (gmt 0)

To split logfiles in editable chunks

Hate to repeat myself ;), but again, Commander.

Macguru

WebmasterWorld Senior Member macguru us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 565 posted 5:02 pm on Aug 8, 2002 (gmt 0)

OK.

But nowadays, I feel like some pain in the sacro illiac aerea reaching for my wallet and pulling the plastic from it. ;)

Any freebee around?

martin

10+ Year Member



 
Msg#: 565 posted 12:32 am on Aug 9, 2002 (gmt 0)

I'm using EditPlus on Windows, it handles everything I throw at it but I'm not sure for 1.8G. A friend of mine told me he used UltraEdit to edit ODP's database ~800MB.

Hint: Use rar, ace, gzip or best bzip2 for compression rather than zip, the difference is really big. ON windows PowerArchiver supports bzip2 compression.

martin

10+ Year Member



 
Msg#: 565 posted 12:48 am on Aug 10, 2002 (gmt 0)

Seeing all those Windows editors... does anybody know of a good Unix/Linux one that can handle very long lines (eg. 20-25K). None works for me.

bcc1234

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 565 posted 9:14 pm on Aug 11, 2002 (gmt 0)

Check out VIM.
It takes getting used to, but once you learn it - you'll love it.

bird

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 565 posted 9:43 pm on Aug 11, 2002 (gmt 0)

Mac, does it really have to be on Windows? Don't you have an OS-X box or linux sitting around?

If all else fails, have a look at cygwin [cygwin.com].
This will bring you the power of the standard unix tools to your Windows box.
Split files: split [hmug.org]
Concatenate files: cat [hmug.org]
Remove bad lines from a file: grep [hmug.org]

Edit files: gvim [vim.org] (available for Windows as well)

  • Maximum line length On machines with 16-bit ints (Amiga and MS-DOS real mode): 32767, otherwise 2147483647 characters. Longer lines are split.
  • Maximum number of lines 2147483647 lines.
  • Maximum file size 2147483647 bytes (2 Gbyte) when a long integer is 32 bits. Much more for 64 bit longs. Also limited by available disk space for the swap-file.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved