Welcome to WebmasterWorld Guest from 54.161.255.61

Forum Moderators: goodroi

Free tools for creating a sitemap

   
4:37 pm on Jul 7, 2011 (gmt 0)

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



I want to convert a site from an ASP based site to a Drupal/PHP based site. The site contains over 500 links but many of them are dynamic links in the form:

discussion.asp?T=123&P=987
newsletter.asp?ID=456
etc.

I need a way to identify all of these links so that I can setup some sort of rewrite to the new equivalents. I was considering [xml-sitemaps.com...] but I don't really want to pay $20 for the unlimited product as I'll only need this one time.

Does anyone know of any other tools that can spider a site and create a large sitemap or list of links on the site?
4:41 pm on Jul 7, 2011 (gmt 0)

WebmasterWorld Administrator brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



WGET can traverse a site no problem, and save the contents locally. I believe you can restrict it to particular filetypes as to avoid images/js etc.

This lays out the possible command you'd use: [linuxjournal.com...]
6:59 pm on Jul 7, 2011 (gmt 0)

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Yeah, I could fairly easily script this, I was just hoping there was something freely available already to save me the time. :)
7:53 pm on Jul 7, 2011 (gmt 0)

WebmasterWorld Administrator brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I bet you could! I think wget would just have a folder with the filenames as your URLs, so minimal scripting is needed really. Just WGET to fetch and sort, and a little script to iterate the filenames.
8:25 pm on Jul 7, 2011 (gmt 0)

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



It's more than just filenames though. For example, there are a ton of links like:

discussion.asp?T=123
discussion.asp?T=456
discussion.asp?T=789

etc., and I need to know what ALL of those links are because I then need to find a way to map each one to a different URL. In other words, each dynamic link needs to map individually to a new location.
8:35 pm on Jul 7, 2011 (gmt 0)

WebmasterWorld Administrator brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



they would all be treated uniquely with WGET, it's one option for you at least.
8:38 pm on Jul 7, 2011 (gmt 0)

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Ah, ok, I misread what you wrote. Ignore my last post. :)
11:50 pm on Jul 7, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Xenu Link Sleuth a freeware app can make sitemaps. http://home.snafu.de/tilman/xenulink.html [home.snafu.de]

The author has a Yahoo! support group [tech.groups.yahoo.com] that has a beta version for huge sites.
12:39 am on Jul 8, 2011 (gmt 0)

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Thanks Hoople. Just gave it a try and it worked great and fast! I still need to dig through the results, but this might be just what I was looking for. Thanks. :)
1:19 am on Jul 8, 2011 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Ann Smarty recently made two blog posts on Xenu's advanced usage.

Worth looking at is the Xenu Wikipedia page Reference section links too.
12:13 pm on Jul 27, 2011 (gmt 0)

5+ Year Member



If you just need it for a once-off you can also use many website analysis utilities (e.g. A1 Website Analyzer), but I think you under all circumstances will need to create a .htaccess file manually redirecting all the old URLs... Unless you are lucky of course and able to reproduce same URL system and content on new CMS, but I think that is unlikely :(
 

Featured Threads

Hot Threads This Week

Hot Threads This Month