homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Code, Content, and Presentation / RSS, ATOM, and Related Technologies
Forum Library, Charter, Moderators: bill & werty

RSS, ATOM, and Related Technologies Forum

How to use RSS to scan 14000 news sources

 12:59 am on Apr 9, 2004 (gmt 0)

I'm always reading how some of the RSS big-wigs are able to scan an insane number of news sources on a daily basis. Do any of you do this? Do you have any pointers for how to amass such a collection of sources? What's a good strategy for filtering all of the information?

I find that it's tough to find sites I would consider reading regullarly in the first place...let alone a large enough collection so that I could have my finger on the pulse of the Internet...or whatever it is these gurus do...



 6:21 pm on Apr 10, 2004 (gmt 0)

It's fairly easy to accomplish--moreso than one would think from the get go. We do it consistently without any problems, but the feeds are not for any other use than our own. Perhaps this is where we save tremendous resources.

I recommend having your own news application custom built rather than going with a service (such as the one in UK). It's cheaper in the long run, and you have total control.

However, I do not think many SEO's are onto this concept yet, but as with anything effective, they will in due time.

Good luck,



 3:05 am on Apr 11, 2004 (gmt 0)

I'm talking about people like Robert Scoble, Joi Ito, Nick Bradbury and the like. They all seem to be using commercially available software to accomplish this.


 4:31 am on Apr 13, 2004 (gmt 0)

There are large collections of RSS sources around. There are lots of free and commercial tools available.

Since the rules of this board (I just got here) seem to strongly discourage any type of self-promotion or linking, I'm not sure how to go about sharing this information.


 8:02 am on Apr 13, 2004 (gmt 0)

Welcome to WebmasterWorld jeffbarr

I'm familiar with a lot of the tools and resource sites, but still wondering how others manage this. For example do you just download other people's opml files or is it a grueling process of accumulating thousands of sources one by one. It seems to me that just having a ton of sources really isn't good. You'd want to qualify those sources somehow, wouldn't you?


 7:34 pm on Apr 24, 2004 (gmt 0)

Thanks for the welcome, Bill.

Some of the feeds are classified using DMOZ or other categorization systems. This simplifies the process. Right now, finding feeds is a manual process, but there are directory sites (PM me for mine) that make use of this info.


 3:49 am on May 10, 2004 (gmt 0)

Hi Bill,

You can use software to scan usenet groups and convert those into RSS feeds. Scraping sites for content isn't that difficult either, all you need is to spider a list of sites on a regular basis really.

[edited by: Woz at 6:56 am (utc) on May 12, 2004]
[edit reason] Tidying up [/edit]


 5:09 am on May 12, 2004 (gmt 0)

Welcome to WebmasterWorld Lionhardt

I guess my conundrum is gathering lists of sites to scan. I'm probably like a lot of you; if it's not mentioned on WebmasterWorld then it probably isn't newsworthy, right? ;) I've been alerted to more major news events via Foo than any other source in recent years...I guess I'll have to keep plugging away on my list then. There don't seem to be many shortcuts to building a quality resource...but there rarely are...

Global Options:
 top home search open messages active posts  

Home / Forums Index / Code, Content, and Presentation / RSS, ATOM, and Related Technologies
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved