joined:Nov 22, 2004
I'm looking for a replacement for the DMOZ directory, anyone have any suggestions?
But before you reply, understand how I was using it:
One of the tools on my site is a "Random Website" tool, that lets you essentially browse a 'random' website. It's a neat tool, useful for sparking brainstorming, and checking network connections where you want to be 100% sure you are not getting cached data.
Well, where does one come up with a list of, in this case, 3.5-4m website? Well, my site simply downloaded the whole DMOZ dump once a week, parsed out all the URLs and built them into an SQL database. It worked very well... until now. DMOZ is dead!
So now I need a new way to aggregate a database of at least a few million websites of no one particular type.
Building my own spider is an interesting idea... but for now, I'd rather pass on that avenue and see if theres anything else out there that is regularly updated/maintained, that I can simply download as needed?
Is it kosher to post relevant links to our own sites on this forum or no? If so I'll post a link so you can see how this thing works. If not, then just trust me it's neat enough to put some effort into saving it if I can.