Link Checker that observes robots.txt and meta tags

[fish_eye]

A few months ago somebody asked for a good link checker both freeware and purchase [webmasterworld.com] and of course there are a few - and one in particular.

Is there one that honors robots.txt and meta tags (FOLLOW/NOFOLLOW specifically)?

FYI: The problem behind my request is that I am using a CMS to generate several thousands of pages (less than 20k) but I only want decent ones indexed. That is, those that have a decent amount of content. So, what I'm doing in the php backend is generating a robots meta tag of "nofollow, noindex" in some cases (most), a "follow, index" when there's a base set of data attributes and images (many) and a "follow, noindex" for pages that are used to browse the site.

My motivation is, to some degree, fear of duplicate content [edit](I mean "being perceived as having several thousand remarkably similar pages" not "people stealing my content")[/edit].

If there isn't one - does anyone want to pay me to develop it ;)

[fish_eye]

txbakers, Should this prehaps be moved to the robots.txt forum?