Welcome to WebmasterWorld Guest from 54.160.163.163

Forum Moderators: bakedjake

Message Too Old, No Replies

Looking for a linux command line app

That lists unique strings only

     

trillianjedi

2:03 pm on Feb 9, 2007 (gmt 0)

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I'm not sure if something like this exists in the standard linux shell. I'm using a combo of GREP and AWK on a logfile tail. I'm looking for a key phrase, but once the line containing that keyphrase is found, I don't want to see it again.

Best explained by example I guess. Here's the kind of combo I'm using:-


tail -f /var/log/mylog.log¦grep "jedi"¦ awk '{ print $13 $9 $10 }'

Everytime Jedi appears in a line, I get that line output. But I only need each unique line once.

Currently I'm getting this for example:-


jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4

Repetitive. What I would like that transformed to is:-


jedi keyphrase1 keyphrase2
jedi keyphrase3 keyphrase4

All duplicates removed.

I can't grep on the keyphrases as I don't know what they are in advance. All I know in advance is that "jedi" will appear in the lines I'm interested in. Whilst that same line is likely to appear in the log again multiple times, I only need to know that it happened once.

Hope that makes sense. Is there something I can do with AWK or GREP or some other app to do this?

Thanks!

TJ

Romeo

3:31 pm on Feb 9, 2007 (gmt 0)

10+ Year Member



You may try to pipe your data to `¦ uniq`, which should eliminate consecutive duplicate lines.

Kind regards,
R.

peterdaly

3:32 pm on Feb 9, 2007 (gmt 0)

10+ Year Member



Pipe into "sort -u" will also do the job.

trillianjedi

4:50 pm on Feb 9, 2007 (gmt 0)

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Thanks guys - excellent! "sort -u" did the trick.

Uniq sounds like it should have done, but didn't - it still sent me out duplicates. Odd?

Anyway, got what I need now - thanks.

marcs

5:23 pm on Feb 9, 2007 (gmt 0)

10+ Year Member



You'd still need to "sort" before you can "uniq", such as :

¦sort¦uniq

That should work fine.

Marc

 

Featured Threads

Hot Threads This Week

Hot Threads This Month