Welcome to WebmasterWorld Guest from 54.205.172.57

Forum Moderators: goodroi

Message Too Old, No Replies

Free tools for creating a sitemap

     
4:37 pm on Jul 7, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 17, 2005
posts: 4993
votes: 14


I want to convert a site from an ASP based site to a Drupal/PHP based site. The site contains over 500 links but many of them are dynamic links in the form:

discussion.asp?T=123&P=987
newsletter.asp?ID=456
etc.

I need a way to identify all of these links so that I can setup some sort of rewrite to the new equivalents. I was considering [xml-sitemaps.com...] but I don't really want to pay $20 for the unlimited product as I'll only need this one time.

Does anyone know of any other tools that can spider a site and create a large sitemap or list of links on the site?
4:41 pm on July 7, 2011 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 30, 2002
posts:4854
votes: 11


WGET can traverse a site no problem, and save the contents locally. I believe you can restrict it to particular filetypes as to avoid images/js etc.

This lays out the possible command you'd use: [linuxjournal.com...]
6:59 pm on July 7, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 17, 2005
posts: 4993
votes: 14


Yeah, I could fairly easily script this, I was just hoping there was something freely available already to save me the time. :)
7:53 pm on July 7, 2011 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 30, 2002
posts:4854
votes: 11


I bet you could! I think wget would just have a folder with the filenames as your URLs, so minimal scripting is needed really. Just WGET to fetch and sort, and a little script to iterate the filenames.
8:25 pm on July 7, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 17, 2005
posts: 4993
votes: 14


It's more than just filenames though. For example, there are a ton of links like:

discussion.asp?T=123
discussion.asp?T=456
discussion.asp?T=789

etc., and I need to know what ALL of those links are because I then need to find a way to map each one to a different URL. In other words, each dynamic link needs to map individually to a new location.
8:35 pm on July 7, 2011 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 30, 2002
posts:4854
votes: 11


they would all be treated uniquely with WGET, it's one option for you at least.
8:38 pm on July 7, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 17, 2005
posts: 4993
votes: 14


Ah, ok, I misread what you wrote. Ignore my last post. :)
11:50 pm on July 7, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 2002
posts:745
votes: 9


Xenu Link Sleuth a freeware app can make sitemaps. http://home.snafu.de/tilman/xenulink.html [home.snafu.de]

The author has a Yahoo! support group [tech.groups.yahoo.com] that has a beta version for huge sites.
12:39 am on July 8, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member fotiman is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 17, 2005
posts: 4993
votes: 14


Thanks Hoople. Just gave it a try and it worked great and fast! I still need to dig through the results, but this might be just what I was looking for. Thanks. :)
1:19 am on July 8, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 2002
posts:745
votes: 9


Ann Smarty recently made two blog posts on Xenu's advanced usage.

Worth looking at is the Xenu Wikipedia page Reference section links too.
12:13 pm on July 27, 2011 (gmt 0)

New User

10+ Year Member

joined:Feb 7, 2006
posts: 29
votes: 0


If you just need it for a once-off you can also use many website analysis utilities (e.g. A1 Website Analyzer), but I think you under all circumstances will need to create a .htaccess file manually redirecting all the old URLs... Unless you are lucky of course and able to reproduce same URL system and content on new CMS, but I think that is unlikely :(
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members