I am looking for a simple asp-script, that can extract URLs from a website. Let's call it a sitemap crawler. I want to be able to feed a domain into the script, and the result should be a list of all pages on the website. It should achieve this by finding internal links on the default page, crawling them, enter the URLs into a database, crawl links on the next page and so on.
Anyone know if such a (free) script exists?
