| Looking for a linux command line app
That lists unique strings only |
trillianjedi
msg:3247508 | 2:03 pm on Feb 9, 2007 (gmt 0) |
I'm not sure if something like this exists in the standard linux shell. I'm using a combo of GREP and AWK on a logfile tail. I'm looking for a key phrase, but once the line containing that keyphrase is found, I don't want to see it again. Best explained by example I guess. Here's the kind of combo I'm using:-
tail -f /var/log/mylog.log¦grep "jedi"¦ awk '{ print $13 $9 $10 }'
Everytime Jedi appears in a line, I get that line output. But I only need each unique line once. Currently I'm getting this for example:-
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4
Repetitive. What I would like that transformed to is:-
jedi keyphrase1 keyphrase2
jedi keyphrase3 keyphrase4
All duplicates removed. I can't grep on the keyphrases as I don't know what they are in advance. All I know in advance is that "jedi" will appear in the lines I'm interested in. Whilst that same line is likely to appear in the log again multiple times, I only need to know that it happened once. Hope that makes sense. Is there something I can do with AWK or GREP or some other app to do this? Thanks! TJ
|
Romeo
msg:3247590 | 3:31 pm on Feb 9, 2007 (gmt 0) |
You may try to pipe your data to `¦ uniq`, which should eliminate consecutive duplicate lines. Kind regards,
R.
|
peterdaly
msg:3247592 | 3:32 pm on Feb 9, 2007 (gmt 0) |
Pipe into "sort -u" will also do the job.
|
trillianjedi
msg:3247681 | 4:50 pm on Feb 9, 2007 (gmt 0) |
Thanks guys - excellent! "sort -u" did the trick. Uniq sounds like it should have done, but didn't - it still sent me out duplicates. Odd? Anyway, got what I need now - thanks.
|
marcs
msg:3247692 | 5:23 pm on Feb 9, 2007 (gmt 0) |
You'd still need to "sort" before you can "uniq", such as : ¦sort¦uniq That should work fine. Marc
|
|
|
