Welcome to WebmasterWorld Guest from 54.196.232.162

Forum Moderators: bakedjake

Message Too Old, No Replies

Looking for a linux command line app

That lists unique strings only

     
2:03 pm on Feb 9, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7242
votes: 0


I'm not sure if something like this exists in the standard linux shell. I'm using a combo of GREP and AWK on a logfile tail. I'm looking for a key phrase, but once the line containing that keyphrase is found, I don't want to see it again.

Best explained by example I guess. Here's the kind of combo I'm using:-


tail -f /var/log/mylog.log¦grep "jedi"¦ awk '{ print $13 $9 $10 }'

Everytime Jedi appears in a line, I get that line output. But I only need each unique line once.

Currently I'm getting this for example:-


jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase1 keyphrase2
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4
jedi keyphrase3 keyphrase4

Repetitive. What I would like that transformed to is:-


jedi keyphrase1 keyphrase2
jedi keyphrase3 keyphrase4

All duplicates removed.

I can't grep on the keyphrases as I don't know what they are in advance. All I know in advance is that "jedi" will appear in the lines I'm interested in. Whilst that same line is likely to appear in the log again multiple times, I only need to know that it happened once.

Hope that makes sense. Is there something I can do with AWK or GREP or some other app to do this?

Thanks!

TJ

3:31 pm on Feb 9, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 28, 2002
posts:505
votes: 0


You may try to pipe your data to `¦ uniq`, which should eliminate consecutive duplicate lines.

Kind regards,
R.

3:32 pm on Feb 9, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 30, 2002
posts:436
votes: 0


Pipe into "sort -u" will also do the job.
4:50 pm on Feb 9, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member trillianjedi is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 15, 2003
posts:7242
votes: 0


Thanks guys - excellent! "sort -u" did the trick.

Uniq sounds like it should have done, but didn't - it still sent me out duplicates. Odd?

Anyway, got what I need now - thanks.

5:23 pm on Feb 9, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Jan 11, 2003
posts:442
votes: 0


You'd still need to "sort" before you can "uniq", such as :

¦sort¦uniq

That should work fine.

Marc