homepage Welcome to WebmasterWorld Guest from 23.22.29.137
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
Forum Library, Charter, Moderators: bakedjake

Linux, Unix, and *nix like Operating Systems Forum

    
Looking for a linux command line app
That lists unique strings only
trillianjedi




msg:3247508
 2:03 pm on Feb 9, 2007 (gmt 0)

I'm not sure if something like this exists in the standard linux shell. I'm using a combo of GREP and AWK on a logfile tail. I'm looking for a key phrase, but once the line containing that keyphrase is found, I don't want to see it again.

Best explained by example I guess. Here's the kind of combo I'm using:-


tail -f /var/log/mylog.log¦grep "jedi"¦ awk '{ print $13 $9 $10 }'

Everytime Jedi appears in a line, I get that line output. But I only need each unique line once.

Currently I'm getting this for example:-


jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4

Repetitive. What I would like that transformed to is:-


jedi keyphrase1 keyphrase2
jedi keyphrase3 keyphrase4

All duplicates removed.

I can't grep on the keyphrases as I don't know what they are in advance. All I know in advance is that "jedi" will appear in the lines I'm interested in. Whilst that same line is likely to appear in the log again multiple times, I only need to know that it happened once.

Hope that makes sense. Is there something I can do with AWK or GREP or some other app to do this?

Thanks!

TJ

 

Romeo




msg:3247590
 3:31 pm on Feb 9, 2007 (gmt 0)

You may try to pipe your data to `¦ uniq`, which should eliminate consecutive duplicate lines.

Kind regards,
R.

peterdaly




msg:3247592
 3:32 pm on Feb 9, 2007 (gmt 0)

Pipe into "sort -u" will also do the job.

trillianjedi




msg:3247681
 4:50 pm on Feb 9, 2007 (gmt 0)

Thanks guys - excellent! "sort -u" did the trick.

Uniq sounds like it should have done, but didn't - it still sent me out duplicates. Odd?

Anyway, got what I need now - thanks.

marcs




msg:3247692
 5:23 pm on Feb 9, 2007 (gmt 0)

You'd still need to "sort" before you can "uniq", such as :

¦sort¦uniq

That should work fine.

Marc

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved