jdMorgan - 2:12 pm on Feb 23, 2007 (gmt 0)

Be aware that using tools/library routines like CURL without changing the default User-agent name to something that gives a clue about who is doing the checking may get you banned from many sites. It's also a good idea to limit the fetch rate from any one given site for the same reason.

A page on your site providing information about your user-agent is a great idea, too.

For a User-agent, follow the major search engines' approach:

Mozilla/5.0 (compatible; DirectoryLinkChecker/1.0; +http://www.MyDirectory.com/linkchecker.html)

Jim