How to use RSS to scan 14000 news sources

[bill]

I'm always reading how some of the RSS big-wigs are able to scan an insane number of news sources on a daily basis. Do any of you do this? Do you have any pointers for how to amass such a collection of sources? What's a good strategy for filtering all of the information?

I find that it's tough to find sites I would consider reading regullarly in the first place...let alone a large enough collection so that I could have my finger on the pulse of the Internet...or whatever it is these gurus do...

[CustomFit]

It's fairly easy to accomplish--moreso than one would think from the get go. We do it consistently without any problems, but the feeds are not for any other use than our own. Perhaps this is where we save tremendous resources.

I recommend having your own news application custom built rather than going with a service (such as the one in UK). It's cheaper in the long run, and you have total control.

However, I do not think many SEO's are onto this concept yet, but as with anything effective, they will in due time.

Good luck,

CF

[bill]

I'm talking about people like Robert Scoble, Joi Ito, Nick Bradbury and the like. They all seem to be using commercially available software to accomplish this.

[jeffbarr]

There are large collections of RSS sources around. There are lots of free and commercial tools available.

Since the rules of this board (I just got here) seem to strongly discourage any type of self-promotion or linking, I'm not sure how to go about sharing this information.

[bill]

Welcome to WebmasterWorld jeffbarr

I'm familiar with a lot of the tools and resource sites, but still wondering how others manage this. For example do you just download other people's opml files or is it a grueling process of accumulating thousands of sources one by one. It seems to me that just having a ton of sources really isn't good. You'd want to qualify those sources somehow, wouldn't you?

[jeffbarr]

Thanks for the welcome, Bill.

Some of the feeds are classified using DMOZ or other categorization systems. This simplifies the process. Right now, finding feeds is a manual process, but there are directory sites (PM me for mine) that make use of this info.

[Lionhardt]

Hi Bill,

You can use software to scan usenet groups and convert those into RSS feeds. Scraping sites for content isn't that difficult either, all you need is to spider a list of sites on a regular basis really.

[edited by: Woz at 6:56 am (utc) on May 12, 2004]
[edit reason] Tidying up [/edit]

[bill]

Welcome to WebmasterWorld Lionhardt

I guess my conundrum is gathering lists of sites to scan. I'm probably like a lot of you; if it's not mentioned on WebmasterWorld then it probably isn't newsworthy, right? ;) I've been alerted to more major news events via Foo than any other source in recent years...I guess I'll have to keep plugging away on my list then. There don't seem to be many shortcuts to building a quality resource...but there rarely are...