Hoople - 7:13 pm on Jul 22, 2011 (gmt 0)
DMOZ.org? The RDF dump of the Open Directory database is available for download. Link and supporting docs [dmoz.org ]
There are a few Firefox plug-ins that allow link harvesting on a page by page basis.
Xenu and some other link checkers can also output a list of URLS as well.