jmccormac - 7:12 pm on Apr 24, 2003 (gmt 0)

don't know how it was done, but it was well done (DMOZ at least). The URL structure is "uniform" and easy to use, no wonder it's used by search engines to accompany results. I'm sure this uniformality is the sort of thing the automated engines love.

For a search engine to get a potential search index from Dmoz, it is really just a case of extracting the links from the RDFs rather than indexing the whole Dmoz. I think there was a discussion on this forum last year that involved someone trying to spider all Dmoz to provide an up to date directory (during the time that the Dmoz RDF were not updated from September 2002 onwards). But it is a very nice way for a search engine to be more than just a search engine by offering a Dmoz fed directory.

Regards...jmcc