How to handle duplicates?

multiple rss feeds displaying duplicate articles

[fyrehawg]

Anyone know how to handle the problem? I have about 8 rss feeds piped into one rss feed however im getting duplicates in my results....most every title is listed twice sometimes three times...anyone know of a reader that can kill the duplicates maybe by compairing titles?

Im sure im not the first person to run into this....just cant seem to find a solution.

Any help would be greatly appreciated.

[bwakkie]

would be good to show some code you are using to combine them

[fyrehawg]

i just got this solved...i used yahoo pipes and fetched 8 feeds i had already set up by searching yahoo news and creating feeds....i then apllied a unique filter in pipes and it works like a charm...if anyone is reading this and having the same problem..check out yahoo pipes...build a pipe and fetch your feeds..then look under the operator tab and apply a unique filter..you can then check for duplicates by title,author,date,url...etc.....be sure and set the filter to your pipes before actually running the pipe...i didnt do this and for a few days i was thinking it wasnt checking for uniques becasue they kept showing up...however its a cache issue...takes about 30 mins for the unique filter to apply itself if it isnt installed on your pipe from creation.

there are ALOT of cool tools and i suspect you could do nearly anything you could think of with rss feeds if you spent some time getting familiar with the options.

i fed the output rss url for my pipes into a the built in feed display in joomla 1.5 and it works great.