Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: not2easy
However, it seems the articles that I link to tend to disappear after a few weeks or so. Any idea why that may be?
I was wondering if I might be able to cache the articles, similar to the way Archive.org does (assuming the site doesn't forbid it with robots.txt) and link to the cache so I know that I'm always linking to a working version of the article.
Altough you can cache (copy) their content you probably shouldn't.
1: You generally violate copyrignt when you cache.. Are you an AHN, AP, Reuters, AFP licensee? Get permission... News organizations register their copyrights, that means statutory damages.
2: If the article gets changed or retracted and then you fail to update then you can be liable for libel or even worse. Lawyers, specifically lawyers for celebrities live for this.
Or is this one of those "I'd probably be right if I followed robots.txt, but I could get dragged through court by lawyers and it wouldn't be worth it" sort of situations?
You can almost guarantee a suit if your site is in anyway commercial in nature or seeks any revenues.
You should read the FAQ and news section.