Welcome to WebmasterWorld Guest from

Forum Moderators: bill & werty

Message Too Old, No Replies

How to handle duplicates?

multiple rss feeds displaying duplicate articles



3:58 am on Dec 17, 2008 (gmt 0)

5+ Year Member

Anyone know how to handle the problem? I have about 8 rss feeds piped into one rss feed however im getting duplicates in my results....most every title is listed twice sometimes three times...anyone know of a reader that can kill the duplicates maybe by compairing titles?

Im sure im not the first person to run into this....just cant seem to find a solution.

Any help would be greatly appreciated.


10:29 am on Dec 19, 2008 (gmt 0)

10+ Year Member

would be good to show some code you are using to combine them


3:13 pm on Dec 19, 2008 (gmt 0)

5+ Year Member

i just got this solved...i used yahoo pipes and fetched 8 feeds i had already set up by searching yahoo news and creating feeds....i then apllied a unique filter in pipes and it works like a charm...if anyone is reading this and having the same problem..check out yahoo pipes...build a pipe and fetch your feeds..then look under the operator tab and apply a unique filter..you can then check for duplicates by title,author,date,url...etc.....be sure and set the filter to your pipes before actually running the pipe...i didnt do this and for a few days i was thinking it wasnt checking for uniques becasue they kept showing up...however its a cache issue...takes about 30 mins for the unique filter to apply itself if it isnt installed on your pipe from creation.

there are ALOT of cool tools and i suspect you could do nearly anything you could think of with rss feeds if you spent some time getting familiar with the options.

i fed the output rss url for my pipes into a the built in feed display in joomla 1.5 and it works great.


Featured Threads

Hot Threads This Week

Hot Threads This Month