homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Free tools for creating a sitemap

 4:37 pm on Jul 7, 2011 (gmt 0)

I want to convert a site from an ASP based site to a Drupal/PHP based site. The site contains over 500 links but many of them are dynamic links in the form:


I need a way to identify all of these links so that I can setup some sort of rewrite to the new equivalents. I was considering [xml-sitemaps.com...] but I don't really want to pay $20 for the unlimited product as I'll only need this one time.

Does anyone know of any other tools that can spider a site and create a large sitemap or list of links on the site?


brotherhood of LAN

 4:41 pm on Jul 7, 2011 (gmt 0)

WGET can traverse a site no problem, and save the contents locally. I believe you can restrict it to particular filetypes as to avoid images/js etc.

This lays out the possible command you'd use: [linuxjournal.com...]


 6:59 pm on Jul 7, 2011 (gmt 0)

Yeah, I could fairly easily script this, I was just hoping there was something freely available already to save me the time. :)

brotherhood of LAN

 7:53 pm on Jul 7, 2011 (gmt 0)

I bet you could! I think wget would just have a folder with the filenames as your URLs, so minimal scripting is needed really. Just WGET to fetch and sort, and a little script to iterate the filenames.


 8:25 pm on Jul 7, 2011 (gmt 0)

It's more than just filenames though. For example, there are a ton of links like:


etc., and I need to know what ALL of those links are because I then need to find a way to map each one to a different URL. In other words, each dynamic link needs to map individually to a new location.

brotherhood of LAN

 8:35 pm on Jul 7, 2011 (gmt 0)

they would all be treated uniquely with WGET, it's one option for you at least.


 8:38 pm on Jul 7, 2011 (gmt 0)

Ah, ok, I misread what you wrote. Ignore my last post. :)


 11:50 pm on Jul 7, 2011 (gmt 0)

Xenu Link Sleuth a freeware app can make sitemaps. http://home.snafu.de/tilman/xenulink.html [home.snafu.de]

The author has a Yahoo! support group [tech.groups.yahoo.com] that has a beta version for huge sites.


 12:39 am on Jul 8, 2011 (gmt 0)

Thanks Hoople. Just gave it a try and it worked great and fast! I still need to dig through the results, but this might be just what I was looking for. Thanks. :)


 1:19 am on Jul 8, 2011 (gmt 0)

Ann Smarty recently made two blog posts on Xenu's advanced usage.

Worth looking at is the Xenu Wikipedia page Reference section links too.


 12:13 pm on Jul 27, 2011 (gmt 0)

If you just need it for a once-off you can also use many website analysis utilities (e.g. A1 Website Analyzer), but I think you under all circumstances will need to create a .htaccess file manually redirecting all the old URLs... Unless you are lucky of course and able to reproduce same URL system and content on new CMS, but I think that is unlikely :(

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved