aakk9999 - 1:18 am on Oct 14, 2011 (gmt 0)

1. NAME - LINK
Screaming Frog SEO Spider [screamingfrog.co.uk...]

2. PURPOSE
Full website crawl, reporting internal and external links, also internal incoming links for each page, produces a report that includes URL, page title, meta tags, response codes and more. Detailed report and excel export.

3. FREE or PAID
Free for the first 500 URLs per site crawl, £99/year full licence

4. WHAT I LIKE
It is a bit like Xenu, but much nicer interface. Allows the crawl to be saved and re-loaded back into the tool. Easy tabbed interface and allows filtering and sorting of results. Like that it also reports on canonical link element (if found on the page), meta refresh, H1, X-Robots tag and some other very useful info. Also like the feature where you can specify custom search of what you want to find in the HTML source code and it will give you a list the URLs where what you specified occurs somewhere in page HTML.

5. WHAT NEEDS IMPROVING
The ability to export sitemap in sitemap.xml format.
Would also like to be able to control the number of threads and control the speed with which the requests hit the server (something that **another tool that gets scrambled letters here** can do) - useful when trying to debug random incorrect HTTP response codes you are seeing in the logs and can never reproduce "on demand".