Hoople - 7:13 pm on Jul 22, 2011 (gmt 0)

DMOZ.org? The RDF dump of the Open Directory database is available for download. Link and supporting docs [dmoz.org ]

There are a few Firefox plug-ins that allow link harvesting on a page by page basis.

Xenu and some other link checkers can also output a list of URLS as well.