Why can't google spider DMOZ and get the directory manually?

You'd think it would be in their best interest....

[born2drv]

Wouldn't it be better to spider DMOZ several times a month and keep the directory fresh rather than wait for the rdf dump? Am I missing something here?

[victor]

Spidering gets you all the pages you can find. It doesn't necessarily get you the correct directory structure.

One example: imagine during the days (ODP allows only one page per second per spider, so googlebot will run s-l-o-w) the pages are being spidered, the top-level cat "Recreation" is renamed "Sports and Recreations". Googlebot will either find it twice under different names or not at all.

[Dreamquick]

Also you have the fact that a single data feed is easier to get and doesn't hammer the DMOZ site like a complete crawl would...

- Tony