1. NAME - LINK
Screaming Frog SEO Spider [screamingfrog.co.uk
Tool to crawl a website and report URLs, response codes, page titles, meta tags, canonical, create sitemap.xml, export URLs as csv and more.
3. FREE or PAID
Free for 500 URLs. £99 + VAT (£118.80) for a yearly licence.
4. WHAT I LIKE
Runs from own PC and has a great easy to use interface. It is very feature rich, it can create sitemap.xml, it can export crawled data with inlinks, outlinks and their anchor text, it can (or not) honour robots.txt, noindex meta when crawling, it can search for a phrase/text in HTML and report all URLs with the phrase/text in it, it can follow redirects, it can crawl or not crawl nofollowed links, etc. Too many features to describe.
For me, it is the essential tool when analysing the site, testing redirects etc. Saves huge amount of time when doing domain migrations and similar activities.
5. WHAT NEEDS IMPROVING
On ocassions, after crawling larger website, if I start to more extensively manipulate the URLs within the tool result set (crawled result set), the tool just dies. I guess it runs out of memory somewhere.
6. WHAT HAS CHANGED (ONLY APPLIES TO OLDER TOOLS)
Lots of new features were added in the last year or so: the ability to control the speed with which to hit the website, the ability to increase the response timeout (great for a slow sites), the ability to export sitemap.xml. Also lots of new data exports were added.
I will second Screaming Frog. Great tool. Used to do the same thing by using Xenu to crawl sites for URLs and then feeding them into my own PHP program. The export to CSV needs some work as exporting data as comma delimited does not import cleanly into Excel on many sites (URL's data wraps over two lines).
[edited by: aakk9999 at 4:35 pm (utc) on Oct 17, 2013]