ODP to 2million yet?

[Brett_Tabke]

Is the ODP to 2million sites yet?

[johncotton]

Yes, 2 Million.
See [biz.yahoo.com]

[Dumpy]

I have several sites using DMOZ. I do not use the world directory, netscape directory, or a couple others. But it takes over four hours to parse the rdf dump into static pages, which creates over 300,000 pages and folders, and requires over 1.5 gigs of space. It is the world's greatest spider trap however! As a side note...each dump contains 4 to 10 errors which create parsing troubles, and each have to be fixed manually, and restart parsing...so a sucessful run can take up to 40 hours! This is why the SE's don't update very often...it's a huge pain!

[Brett_Tabke]

Thank John, I thought I'd read that somewhere, but couldn't find the story in my news archives.

Dumpy, I always figured someone would come up with routine to pull it straight out of the zip file on the fly. (c or ml of course). Load a whole compressed version of the thing into memory and do the search right out of there.

I dl'd the rdf dump again yesterday. I can't believe how many dupes they are allowing these days. There are some domains with hundreds of listings - that isn't right.

[Geniac]

The total number of sites, categories and editors are listed at the bottom of the main page at dmoz.org

Duplicates.. Yes, there are duplicates, especially between Regional and the rest of the directory. Many instances of overlapping categories have been eliminated over time.

> There are some domains with hundreds of listings - that isn't right.

There is a list of domains that are allowed to have multiple sub-pages listed in different categories.
All the free homepage providers are on the list, for example.

Please, people, if you have a complaint or query about ODP editors or categories, give some detail. The ODP editors here cannot help you if you simply say "some domains", "these editors", "the category my site is listed in", etc.

We are here to help. We are here to prove we are humans that err.

[Brett_Tabke]

Appreciate you stopping in Geniac. Funny, I'd never noticed the stats at the bottom of the main page.

I don't think anyone had specific complaints here, just in general what we'd been seeing.