Free tools for creating a sitemap

[Fotiman]

I want to convert a site from an ASP based site to a Drupal/PHP based site. The site contains over 500 links but many of them are dynamic links in the form:

discussion.asp?T=123&P=987
newsletter.asp?ID=456
etc.

I need a way to identify all of these links so that I can setup some sort of rewrite to the new equivalents. I was considering [xml-sitemaps.com...] but I don't really want to pay $20 for the unlimited product as I'll only need this one time.

Does anyone know of any other tools that can spider a site and create a large sitemap or list of links on the site?

[brotherhood of LAN]

WGET can traverse a site no problem, and save the contents locally. I believe you can restrict it to particular filetypes as to avoid images/js etc.

This lays out the possible command you'd use: [linuxjournal.com...]

[Fotiman]

Yeah, I could fairly easily script this, I was just hoping there was something freely available already to save me the time. :)

[brotherhood of LAN]

I bet you could! I think wget would just have a folder with the filenames as your URLs, so minimal scripting is needed really. Just WGET to fetch and sort, and a little script to iterate the filenames.

[Fotiman]

It's more than just filenames though. For example, there are a ton of links like:

discussion.asp?T=123
discussion.asp?T=456
discussion.asp?T=789

etc., and I need to know what ALL of those links are because I then need to find a way to map each one to a different URL. In other words, each dynamic link needs to map individually to a new location.

[brotherhood of LAN]

they would all be treated uniquely with WGET, it's one option for you at least.

[Fotiman]

Ah, ok, I misread what you wrote. Ignore my last post. :)

[Hoople]

Xenu Link Sleuth a freeware app can make sitemaps. http://home.snafu.de/tilman/xenulink.html [home.snafu.de]

The author has a Yahoo! support group [tech.groups.yahoo.com] that has a beta version for huge sites.

[Fotiman]

Thanks Hoople. Just gave it a try and it worked great and fast! I still need to dig through the results, but this might be just what I was looking for. Thanks. :)

[Hoople]

Ann Smarty recently made two blog posts on Xenu's advanced usage.

Worth looking at is the Xenu Wikipedia page Reference section links too.

[spiritparse]

If you just need it for a once-off you can also use many website analysis utilities (e.g. A1 Website Analyzer), but I think you under all circumstances will need to create a .htaccess file manually redirecting all the old URLs... Unless you are lucky of course and able to reproduce same URL system and content on new CMS, but I think that is unlikely :(